Emilee Rader and Anjali Munasinghe. “Wait, Do I Know This Person?”: Understanding Misdirected Email. CHI 2019, Glasgow, UK, May 2019. DOI: 10.1145/3290605.3300520

1 Survey Overview

1.1 Data Collection

1.1.1 When did the survey take place?

Data collection for the survey took place starting on 2018-03-20 and ending on 2018-04-02.

1.1.2 How many responses were collected?

The survey was started by 943 potential respondents. After data cleaning, there were 380 respondents in the dataset.

1.1.3 How long did it take to complete the survey?

The survey took an average of 12.94 minutes for respondents to complete, including the consent and screening questions. The maximum completion time was 80.75 minutes, and the minimum was 3.25 minutes. Respondents had 24 hours from the time they began the consent form to complete the survey, and were told this in the instructions.

Completion Time Descriptives (in minutes)
Min 3.2
Median 10.5
Max 80.8
M 12.9
SD 9.9

1.2 Sample

1.2.1 How many respondents were ineligible vs. excluded?

Respondents were recruited by Qualtrics using their panel service, with quotas for gender (50% men and 50% women) and age (18-29: 25%, 30-49: 38%, 50-64: 21%, 65+: 15%). The age quota was based on information from the Pew Research Center’s Internet/Broadband Fact Sheet from Feb. 5, 2018 about the age distribution of US adults who use the internet, and data from the US Census Bureau’s 2016 American Community Survey about the age distribution of the United States population.

The Pew results tell us what proportion of different age groups use the internet (e.g., 66% of US adults who are 65+ are internet users). But that doesn’t tell us how many people in the US are 65+ years old, so we can’t calculate what proportion of our respondents should be 65+ just using the Pew data. From the Census ACS data, we know that in 2016, the most recent year for which data had been published at the time this research was conducted, 15% of the US population is 65+. So now we know that 66% of the 15% of the US population who are 65+ use the internet. The ACS also includes the information that there are about 323,127,515 people in the US, so using information from both sources we can calculate reasonable target percentages for survey recruiting.

Eligible respondents also were older than 18 and younger than 120 years old, and indicated that they had at least one email account on a popular free email service. The text of the question was, “Do you currently regularly use at least one email account from a popular free email service provider like Gmail, Outlook Mail or Hotmail, Yahoo! Mail, AOL Mail, etc.?”

Respondents who did not consent and who did not meet the age, gender, and email account screening criteria were excluded after declining consent or after providing an ineligible answer to one of the screening questions.

1.2.2 For what reasons were eligible respondents excluded?

  • duplicate name and IP address: During data cleaning, we discovered that two respondents had taken the survey multiple times, based on the IP address associated with their responses, the first and last names they provided, and their answers to demographic questions being duplicated across multiple completed surveys. We kept the first completed survey from each respondent in the dataset, and removed the duplicates.

  • failed attention check or finished too quickly: The survey included an attention check question and a check for finishing the survey too quickly. The check for “speeding” excluded responses from respondents who finished the survey in 1/3 the median time to complete, based on the “soft launch” of the survey. This works out to 3.54 minutes. The attention check consisted of asking respondents for their username (only the part before the @ sign) from their email address on a popular free email service, and then asking again for the same username later in the survey. If these did not match, they were directed to the end of the survey without being allowed to finish it. Respondents were told in the consent form and instructions that there would be “questions and other methods designed to ensure you are paying attention”. And, the first time they entered their username, they were asked to enter it and then re-enter it, and weren’t allowed to proceed until the usernames matched, to ensure that people weren’t kicked out of the survey due to a typo in the username.

  • poor quality name or username: The survey included a question asking particpants to provide their first and last name and email username only (the part before the @ sign) for the free email service they belonged to. The survey included the instruction that this information would be used only for using similarity metrics to compare their name to their username, and would then be deleted. (“This information will be used to determine how similar your username is to your name, and will be deleted as soon as the comparison is complete.”) The name and username responses were inspected by hand to identify any that did not follow the instructions, and these responses were excluded. Some examples of this are responses that were only initials, gibberish (e.g. “ggvv”, “Th”, “dslkdasklklsd”), and responses that were not names (e.g. “boobs”, “none”, “I don’t know”, “refused”).

  • poor quality short answer: The survey included two short answer questions, one asking them to explain why they would pick the new username that the entered in a previous question (variale name: “why_username”“), and the other asking them to describe an example of a wrong email that they had received (variable name:”wrongemail_example“). These responses were examined by hand and those that did not answer the question were excluded. We retained (did not exclude) cases with responses that contained an answer to the question, but then added spaces or punctuation or other text to meet the length requirement of the text entry field. Here are a few examples of excluded responses to the”why_username“” question:

    • “I wont be creating a new account, unless i want to. Man, bump these sentence. Ill make this short if i want ro.”
    • “Idk what to say and ……………………………..hsjsjsjsjsjjsnxjsjsjsjjddjndjsjsjsjdjxnxjdjsjxjjdjdn. Jenxjsnskxisnxjdj”
    • “Nothing at all don’t know nothing at all don’t know nothing at all don’t know nothing at all don’t know nothing at all don’t”

  • “good or full” familiarity with fake word: The internet literacy question consists of a set of internet-related terms. Respondents were asked to specify their level of familiarity with the terms, ranging from “None” to “Full” familiarity. One of the words in the set was a fake word that respondents should not be familiar with. Respondents who said they had “Good” or “Full” familiarity with the fake word were excluded after completing the survey.

  • incomplete survey: Respondents who started but didn’t finish the survey at the time the survey was closed and the dataset was downloaded from Qualtrics.

Here is the number of responses excluded for each reason:

excluded n
did not consent or ineligible 229
failed attention check or finished too quickly 208
attention2 fake word good or full 59
poor quality name or username 30
poor quality short answer 28
incomplete survey 4
duplicate name and IP address 3
over quota 2

1.3 Respondent Demographics

The text of the survey questions about respondent demographics are included in the Demographics Block section of this document.

1.3.1 Age, Gender, Ethnicity

The average age of respondents was 44.99 (SD=16.81). Age ranged from 18 to 85. There were 196 women 181 men who participated in the study.

Age Descriptives (in years)
Min 18
Median 43
Max 85
M 45
SD 17

A large majority of respondents were white. Note that the ethnicity question allowed respondents to “choose all that apply”; in the first “ethnicity” table below “TRUE” indicates that the respondent checked the box, and NA indicates that they did not.

ethnicity counts
white 308
hispanic 21
black 47
asian 14
native_am 7
middle_eastern 0
pacific 2
other 2

1.3.2 Region, Income, Education

Here are the characteristics of the sample in terms of region of the country, income, and education level:

1.3.3 Internet Literacy

The internet literacy variable consists of questions that are based on the Web Use Skills survey reported in Hargittai and Hsieh (2011). Respondents were asked the following question about the internet-related terms: “How familiar are you with the following Internet-related terms? Please rate your understanding of each term below from None (no understanding) to Full (full understanding)”.

We created a composite variable by averaging the self-reported responses for each respondent across the eight internet literacy items. The overall mean was 2.59 (SD = 0.89, median = 2.62). Cronbach’s alpha (a measure of internal consistency): 0.83.

The responses to each item are shown in the graphs below:

2 Survey Questions

Note: This section presents the complete survey instrument, along with descriptive statistics for each question. Formatting and pagination of the questions in the actual survey as it was administered differs from how the questions are presented in this document. Text shown in italics was not part of the survey.

2.1 Screening Block

The three questions below were asked after respondents had consented, but before starting the actual survey, to determine eligibility to participate. Respondents were ineligible if they reported being younger than 18 or older than 120 years old, and if they said “No” or “I’m not sure” to the email account question. The age and gender questions were used to meet the survey recruiting quotas.

  • What is your age in years? [fill in the blank]

  • What is your gender? [Man, Woman, Other (fill in the blank), Prefer not to disclose]

  • Do you currently regularly use at least one email account from a popular free email service provider like Gmail, Outlook Mail or Hotmail, Yahoo! Mail, AOL Mail, etc.? [No, Yes, I’m not sure]

2.2 Email Accounts Block

Instructions: The survey should take approximately 15 minutes to complete. You have up to 24 hours to finish the survey, starting from the time you started reading the consent form on the previous page. Please note that the study includes questions and other methods designed to ensure that you are paying attention. If you do not pay careful attention to every question while you are completing the survey, you will be directed to the end of the survey without being able to complete it.

2.2.1 Reasons for creating a new email account

This question is based on Bentley et al.’s CHI 2017 paper “If a person is emailing you, it just doesn’t make sense”: Exploring Changing Consumer Behaviors in Email.

For which of the following reasons have you ever created a new email account? Please select all that apply:

  • To have a more professional sounding username
  • Old account getting too much spam
  • Didn’t want to give “real” email address to a business
  • Didn’t want to give “real” email address to a person
  • To sign up for a service that requires a specific email provider such as Google Docs (gmail) or Xfinity (Comcast)
  • Wanted to keep personal emails separate
  • Problems with old account (locked out, deleted for non-use, etc.)
  • To use for giving out to websites, such as online shopping sites
  • To try something new
  • Graduated from school and needed a new email address
  • Started a new job and got a new email address from my employer
  • Created a new account for a specific project, business or hobby
  • Other (please fill in the blank)
  • None of the above
reasons count percent
Wanted to keep personal emails separate 142 37.4
Old account getting too much spam 135 35.5
To have a more professional sounding username 94 24.7
To use for giving out to websites, such as online shopping sites 86 22.6
To sign up for a service that requires a specific email provider 82 21.6
Problems with old account 74 19.5
Didn't want to give 'real' email address to a business 68 17.9
To try something new 59 15.5
None of the above 46 12.1
Created a new account for a specific project, business or hobby 46 12.1
Didn't want to give 'real' email address to a person 42 11.1
Started a new job and got a new email address 40 10.5
Other reason 16 4.2
Graduated from school and needed a new email address 15 4.0

2.2.2 How many email accounts does each respondent have?

In the space provided below, please list the email accounts that you currently frequently use, starting with the email account you use the most often. If you use several email accounts about equally, please list up to five of them. Please do NOT enter the complete email address for each account. Instead, give each email account a nickname or a short phrase that describes it. For example, “personal email I’ve had the longest”, “work email”, “school email”, etc.

  • Email account #1 (1) [fill in the blank]
  • Email account #2 (2) [fill in the blank]
  • Email account #3 (3) [fill in the blank]
  • Email account #4 (4) [fill in the blank]
  • Email account #5 (5) [fill in the blank]

Displayed only if the respondent entered something into all five blanks in the previous question:
Do you have more email accounts than the five you listed in the previous question? [No, Yes, I’m not sure]

More than Five Accounts n
No 17
Yes 8
NA 355

2.3 Email Accounts Loop Block

Note: These questions were repeated once for each email account listed above. While all respondents filled in at least one blank, not all respondents filled in more than that. The NAs are removed from the graphs below.

2.3.1 How long have respondents had each account?

How long have you had the email account “${lm://Field/1}”?

  • Less than one year
  • 1-2 years
  • 2-4 years
  • 4-6 years
  • 6-10 years
  • More than 10 years
  • I’m not sure

2.3.2 How did respondents choose their usernames?

When you created the email account “${lm://Field/1}”, how did you choose the username for that account? (The username is the part of the email address before the @ sign.) Select the response below that most closely matches your experience:

  • I created the exact username I wanted.
  • A username was assigned to me, and I kept it.
  • A username was assigned to me, but I changed it later.
  • The system suggested a username to me when I created the account, and I accepted it.
  • Most of the usernames I tried to create were already taken, so I ended up with this one.
  • I don’t remember.
  • None of the above.

2.3.3 How early on did respondents create their email accounts?

At the time you created your “${lm://Field/1}” account, about how soon was it after the email service first became available? Please choose the response that best represents what you remember:

  • Very soon after
  • Somewhat soon after
  • A while after
  • Somewhat long after
  • A long time after
  • I don’t know

2.4 Names and Naming Block

2.4.1 Characteristics of respondents’ names

Instructions: The following question asks for three pieces of information, your first name, your last name, and ONLY the username (the part before the @ sign) of your email account from a popular free email service provider like Gmail, Outlook Mail or Hotmail, Yahoo! Mail, or AOL Mail that you use most often. This information will be used to determine how similar your username is to your name, and will be deleted as soon as the comparison is complete.

Note: descriptive statistics about respondents’ names were calculated, and edit distance comparisons between names and usernames were also computed. Then the raw name data were deleted from the dataset. Only the descriptive statistics were preserved, to protect respondents’ privacy.

Please enter the requested information below, and make sure to check carefully for typos:

  • First Name [fill in the blank]
  • Last Name [fill in the blank]
  • Email Username [fill in the blank]

Please re-enter your First Name, Last Name, and the Email Username that you entered above:

  • First Name [fill in the blank]
  • Last Name [fill in the blank]
  • Email Username [fill in the blank]

First name and last name length in number of characters:

Name Length (nchars) Min Median Max M SD
firstname_lower_nchars 3 5 11 5.6 1.6
lastname_lower_nchars 2 6 14 6.4 1.8

2.4.2 Respondents’ self-report estimate of how common their name is

In your opinion, how common or uncommon is your FIRST name?

  • Extremely common
  • Moderately common
  • Slightly common
  • Neither common nor uncommon
  • Slightly uncommon
  • Moderately uncommon
  • Extremely uncommon

In your opinion, how common or uncommon is your LAST name?

  • Extremely common
  • Moderately common
  • Slightly common
  • Neither common nor uncommon
  • Slightly uncommon
  • Moderately uncommon
  • Extremely uncommon

2.4.3 Characteristics of respondents’ usernames, and suggested new username

Please imagine that you are creating a new email account on a brand new email service that currently has very few users, and you can have any email username that you want. What username would you choose? Please enter it below: [fill in the blank]

Username length (nchars), and whether usernames have digits or special characters:

variable Min Median Max M SD
username_parsed_nchars 3 11.0 32 11.57 4.73
new_username_parsed_nchars 2 9.5 25 9.96 3.85
username_digits 0 1.0 1 0.52 0.50
new_username_digits 0 0.0 1 0.42 0.49
username_specials 0 0.0 1 0.11 0.31
new_username_specials 0 0.0 1 0.06 0.23

2.4.4 Why did people say they’d choose a particular username?

Note: the text in this section describes the content analysis of respondents’ reasons for choosing their usernames. The survey question is presented below. The remaining text in this section was not part of the survey.

Survey question: Please explain why you would choose the username “${choose username/ChoiceTextEntryValue}” if you were creating an account on a brand new email service that currently has very few users? Your answer must be at least 100 characters long, which is about 2-3 sentences. [fill in the blank]

We made a first pass through the responses and developed a set of categories of reasons why they would choose a particular username. These “reasons” reflect concerns they had, potential future uses of their email account, and characteristics they wanted their username to have. Then, two coders coded the responses using these categories.

Below are the codes, and the inter-rater reliability (Fleiss’ kappa):

Name and Birthdate

  • Own Name: Their own name (kappa = 0.91)
  • Own Birthdate: Their own birth date, year, etc. (kappa = 0.91)
  • Own Nickname: Their own nickname (kappa = 0.91)

Life Characteristics

  • Friend Name: The name of a friend or family member (kappa = 0.59)
  • Location: A location, neighborhood, zip code, phone number (kappa = 0.87)
  • Proper Noun: A proper noun that is the name of something (like a pet, fictional character, sports team or band, etc.) (kappa = 0.55)

Personal Meaning

  • Meaningful: Some kind of personal meaning (like about their life, activities, interests or personality) (kappa = 0.53)
  • Fits Me: I just like it, it just fits me, it just came to mind, etc. (kappa = 0.49)
  • Catchy: It’s catchy, creative, fun, etc. (kappa = 0.72)
  • Unique: It’s unique, uncommon, unusual, unlikely, etc. (kappa = 0.77)

Future Uses

  • Memorable: Easy to recognize or remember (kappa = 0.81)
  • Same ad Before: It’s the same username as they used in the past, or on other systems (kappa = 0.7)
  • Professional: Professional or business related (kappa = 0.55)
  • Security: Mentions privacy, identity theft, hacking, anonymity, spammers (kappa = 0.79)

None of the above (mutually exclusive) (kappa = 0.43)

The table and graph below show counts for each of the codes which had inter-rater reliability above 0.5, and on which either one of the coders said yes, that motivation was present.

code n
Memorable 198
Own Name 114
Meaningful 112
Unique 65
Same as Before 56
Professional 39
Proper Noun 34
Security 27
Own Birthdate 27
Own Nickname 27
Catchy 19
Friend Name 14
Location 9

The categories were not mutually exclusive, except where indicated in the list of categories, above (fake response or none of the above). The categories were further grouped into four high-level sets of categories as shown in the bar chart. The tables below present counts of how many distinct respondents gave reasons that fell into each category.

Future Uses n
TRUE 238
FALSE 142
Name and Birthdate n
TRUE 144
FALSE 236
Life Characteristics n
TRUE 56
FALSE 324
Personal Meaning n
TRUE 171
FALSE 209
Name and Birthdate, Personal Meaning n
TRUE 307
FALSE 73

2.4.5 What respondents consider when creating a new username

What is the most important thing for you to consider when creating a new email username? Please drag the following options to put them in order according to how important they would be for you:

  • My new username should be easy for others to remember
  • My new username should be similar to my real name
  • My new username should be the same as my username on other accounts
  • My new username should be unique compared to other people’s email usernames
  • My new username should represent some aspect of my interests or personality
  • My new username should sound professional
  • My new username should be hard for spammers to guess
Question Min Median Max M SD
My new username should be hard for spammers to guess 1 4 7 4.1 2.0
My new username should sound professional 1 5 7 4.5 2.0
My new username should be similar to my real name 1 4 7 3.9 2.0
My new username should be easy for others to remember 1 4 7 3.7 1.9
My new username should represent some aspect of my interests or personality 1 4 7 3.9 2.0
My new username should be the same as my username on other accounts 1 4 7 4.3 2.0
My new username should be unique compared to other people's email usernames 1 3 7 3.6 2.0

Bar charts of the ranks assigned to each item: