VDC Research: Enterprise & Government Mobility Market Blog

Behind the Research: 10 Tips for Survey Design

This post is the first in an upcoming short series of blogs called Behind the Research. Our Behind the Research series is intended to provide existing and potential clients with targeted insights into our some of our key methodological approaches pertaining directly to our data.  While this series is by no means exhaustive, our Analysts will lend their expertise and provide tips on methodological topics of interest and relevance.  If you have a topic which you would like to see covered in a future issue of Behind the Research, feel free to e-mail your suggestion to [email protected] or Tweet your suggestions to @VDC_Research.

Polls and surveys have the ability to provide significant insights into many areas of research and can often reveal hidden trends.  Whether used in politics or market research, surveys provide analysts with vital data for forecasts, predictions, and cross-cutting analysis of their desired topic area.  Many of VDC’s reports, whitepapers, and blog posts contain data provided from our custom designed surveys. 

The nature of VDC surveys allow respondents to provide information from an anonymous, protected vantage point, giving them the ability to respond freely with information regarding markets, business structures, and other strategic details.   However, before leveraging the capabilities of any survey, it should pass through rigorous planning stages to ensure that it is not only effective, but will also yield the best possible environment for accurate results.  This design process further solidifies internal and external validity, ultimately reducing Type I and Type II errors in the following analytics process.  While there are many strategies involved in survey design, this blog post will attempt to highlight 10 critical tips to keep in mind when designing your own survey.

1. Outline First:  Overall, when designing a survey, it’s best to create a rough outline of how your survey will flow, the logic, and the block organization before programming it into your desired survey platform (Qualtircs, Survey Monkey, etc.).  This can save a lot of time during the revision process when changing blocks or re-order questions. 
2. Create Fair Questions:  Know your audience and design questions to elicit a fair answer.  A biased question will give you biased answers.  Using leading questions or answers which clearly favor one response, will often result in heavily skewed and unreliable data.  When using Likert scales try to avoid terms like “Always” and “Never” in favor of terms like “Not Very Much” or “Extremely”.
3. Limit Response Options:  Consider limiting the number of response options or design close-ended questions.  If you have a multiple choice question which asks participants to “select all that apply” and give them a list of ten (or more) items to assess, you will likely receive watered down data points and might miss trends due to aggregation.  In the case described above, respondents will generally select quite a few choices, but you can combat this through response control.  If participants can only select 3 or 5 options, they will provide more accurate responses. 
4. Beware of Respondent Fatigue:  If you have a HUGE question, sometimes it may be best to break it into a series of shorter ones.  Participants will only give each question a certain amount of time before they become tired and move on.  So, if you have a question followed by a long list of Likert scale items, or one which requires heavy reading, participants will probably tune-out about half way through due to fatigue.  You can combat this phenomenon by making several shorter questions instead of one long question.  Limiting response options will also help prevent respondent fatigue.
5. Attention Checks:  Always ask an attention check question.  This will allow you to quickly purge any respondents who fail this basic test in the data cleaning phase of your project.  You don’t have to hide this question; if people are not paying attention, then you don’t want them in your data anyways.  Be sure to insert the attention check question(s) in the middle or end of your survey to catch satisficers or flatliners.  You’ll be surprised at how many people actually fail a question as simple as this: 
6. Page Breaks:  Be sure to put a page-break after lengthy questions or after two smaller back-to-back questions.  Too much information or too many questions on any one page can easily overwhelm respondents and increase fatigue.
7. Organize Questions:  Put questions with the same theme or topic into the same block (organizational structure).  This way, respondents answer all the pertinent questions regarding one topic at the same time.  By focusing all of their attention onto one general topic at a time, they are able to provide more accurate responses.  This organization will significantly help when using survey logic tools as well.
8. Use Survey Logic: Use “skip logic” and “display logic” to map out your survey.  This way, participants will only receive specifically applicable questions.  Skip logic will allow you to skip a question if a respondent does not meet certain criteria.  Display logic will allow you to only display a question if a respondent meets specific criteria.  While these tools many not be needed on simple surveys, on a long more complex survey, logic coupled with organized questions can save a lot of programming time.
9. Fore Validation:  Remember to force respondents to answer questions which are essential to your data or which are important to the general flow of the survey.  Any question which is dependent on skip or display logic should receive this treatment.
10. Test Your Survey:  ALWAYS test your survey!  You essentially want to try to break your survey by being the worst respondent possible.  This way you can catch any logic issues, organizational problems, or other errors which might be lurking before you launch to your actual target audience.  Also, feel free to give it to co-workers and let them try to break it too!

Cameron Roche received a dual B.A. in psychology and government & law from Lafayette College and his M.A. in political science, specializing in political psychology, American politics, and polling/survey methodology, from the University of Massachusetts, Amherst.  He previously served as Assistant Director for UMass Poll and as a Research Assistant for the Cooperative Congressional Election Survey (CCES).  Be sure to look for future Behind the Research posts addressing other methodological aspects of survey work and data cleaning.  The author can be contacted directly at [email protected] or via his Twitter @Cam_Roche. 

The Postal-Courier Industry’s Looming Technological Challenge

The postal-courier industry, in the midst of market changes, will need to grapple with a significant technological challenge looming on the horizon. The delivery of goods is an expensive proposition that requires significant capital investments and a large supply of labor. Technology, in the form of handheld devices and delivery software, works to create an efficient process that enables companies to minimize costs and maximize revenues. The handhelds used by delivery personnel include a number of important capabilities—bar code scanning, messaging, GPS—that help facilitate workflows. From enabling track and trace to obtaining proof of delivery, the current use-cases for these devices are plentiful. As the expectations of consumers grow and their desire for greater personalization increases; postal-courier organizations will need to heavily rely on these devices to transfer real-time data between all parties involved in a transaction. Moreover, postal-courier organizations attempting to expand their service offerings to offset declining mail volumes and differentiate themselves from the competition will employ handheld devices and new applications to complete additional business processes. To this end, upgrading to a new, more modern operating system helps organizations expand into auxiliary, customer-facing services.

Today’s Operating System

While several postal-courier organizations have chosen smartphones as their delivery device, the vast majority have opted for ruggedized enterprise mobile computers. Over 80% of these devices run on legacy Windows Embedded CE and Windows Mobile 6.x platforms. These legacy systems are particularly well-suited for enterprise devices and applications due to their customizable nature and strong ecosystems of solution providers developing applications. Moreover, comprehensive lifecycle support from Microsoft and OEM partners provides further value-add for these platforms. However, Microsoft ended meaningful support for Windows Mobile 6.x in January 2015, and support for Embedded CE will end shortly; with Microsoft only providing security patches for the platforms through 2020. As a result, over the course of the next few years, postal courier organizations will need to transition to a new operating system. 

 

In theory, the logical next step would be to transition to the most recent iteration of Windows without needing to implement any serious changes. However, the situation is far more complicated than that; leaving many organizations debating their next steps.

The Path Forward

With the deadline of 2020 fast approaching, companies must begin evaluating their options regarding hardware, operating system, and software. In choosing an OS for next generation enterprise handheld devices, the decision comes down largely to the consumer heavyweight, Android, or the enterprise incumbent, Microsoft. However, with Microsoft having failed to deliver a desirable operating since the release of Windows Embedded Handheld 6.5 in 2011; some in the industry have moved to the Android operating system with its modern features and capabilities.

Android does not represent a perfect solution. Designed for the consumer space, the operating system lacks many of the security and management features of Windows as well as the substantial base of enterprise developers and OEMs supporting it. Nonetheless, Android’s enterprise user-base has grown steadily over the past few years, coinciding with improvements in the OS’s enterprise functionality. All leading rugged OEMs now offer an Android option, and several view this operating system as the way forward for the industry. Moreover, many ISVs that previously supported legacy Windows systems have shifted their R&D focus to Android (and iOS). This growing portfolio of Android devices and enterprise applications serves as justification for many to switch to this OS in light of Microsoft offering no suitable alternative.

Microsoft’s hopes to regain its market share, or at the very least stop the flow of users away from Windows, lie with its new OS for enterprise handheld devices—Windows 10 IoT Core. Postal-courier organizations, specifically those in the United States, have a pent up demand for a Windows solution, albeit having been burned by Microsoft in the past. Failing to deliver a Windows 7 solution and releasing an unsuitable solution in Windows 8; confidence among those in the space is waning. Windows 10, with its cross-device functionality and enterprise features could meet this demand. However, the release continues to be delayed, incentivizing those in need of a more immediate upgrade to switch to Android. Continued delays, or the release of an incomplete product, would further spur Android adoption, essentially sealing Microsoft’s fate in the rugged enterprise space.

What about the Apps?

Devices and operating systems play an important role in the completion of business processes, but applications/software are the last piece of the puzzle. Most organizations either develop their applications internally or outsource the process, but either way, the finished applications provide the key to performing tasks in the field. Unfortunately, postal-courier organizations planning an OS transition will also need an application-transition plan. For regardless of OS choice, organizations will need to recode old applications or develop new ones.

Windows 8 and 10 are not backwards compatible with legacy Windows systems, and a move to Android or iOS will similarly require recoding. This process, in most cases, will be expensive and cumbersome. If the company has the technological resources, they may complete the project in-house; however, most organizations will likely need to outsource at least some of the project. There are a number of tools on the market, such as Xamarin or iFactr, which enable cross platform development, and essentially provide organizations with a bridge to their next OS. Microsoft is also providing hooks from Visual Studio (VS) to the Xamarin platform making it easier to leverage existing .NET and C# skillsets to support development on Android or iOS. Moreover, we expect many in the industry to use this period as an opportunity to modernize applications that have been in service for decades.

The Bottom Line

Handheld devices play an instrumental role in the postal courier market now and moving forward. The next few years mark a period of mass transition of the likes never seen in the postal courier industry, and the lack of leadership by Microsoft creates an environment of uncertainty. However, this period also marks a moment of opportunity for organizations to modernize their systems and applications. The path taken today will largely dictate the quality of enterprise mobility capabilities tomorrow.

Written by Matthew Hopkins and David Krebs.  For more information, be sure to review our forthcoming full postal-courier report, set to be released this month (November), or contact us at [email protected].  The author can be contacted directly at [email protected].