Knowledge Advisory Group

Trina Willard

4 Smart Tips for Hiring an Evaluator

birdYou know that your organization needs an evaluator…but how do you go about it? Hiring the right evaluator can save your organization a considerable amount of time, effort, and money. For many organizations with limited internal capacity, an external evaluator can often produce more accurate, useful, and credible information in a time-efficient manner, as compared to information generated by a staff member without specific evaluation expertise. Continue reading

Great Online Resource: ICPSR

icpsrThe ICPSR website is a great place to source a variety of information, reports, and tools for finding data to use for evaluation.

ICPSR is the Inter-university Consortium for Political and Social Research. It’s a global leader in data stewardship, providing a wealth of data resources and responsive educational opportunities. They store, curate, and provide access to scientific data so that others can use it. Continue reading

The Top 10 Reasons Why I Enjoyed Evaluation 2014

Evaluation 2014 | Knowledge Advisory GroupI recently traveled to Denver, CO for Evaluation 2014, the annual conference of the American Evaluation Association.  It was an incredibly fun and informative trip, filled with networking, new techniques, and innovative resources.  Hurrah for continuous improvement!

Last year, I reflected on the event in the style of David Letterman’s top ten list.  I’ve decided to repeat that effort this year (albeit, a bit late for the after-party). Perhaps it will be a new tradition! Without further fanfare, I offer:

The Top 10 Reasons Why I Enjoyed Evaluation 2014

  1. Dude. It’s Denver. We’re a mile high and there are evaluators…everywhere…with fancy ribbons! It’s like some kind of geeky heaven!


  1. Social network analysis is cool, even if that workshop full of people was righteously hot. You may operationally define “hot” amongst yourselves.


  1. TIG Madness! I’ve signed on as a Facebook contributor for the Independent Consulting TIG and I can’t wait to get started!  I also joined the NPFTIG dinner at the Appaloosa Grill on Thursday night. I’d encourage your attendance at this event at next year’s conference – this is a truly great group.


  1. Drinks with these smart, fun, marvelous human beings. What happens in Denver stays in Denver. ‘Nuff said.


  1. There are data viz goddesses all over the place. Hey Stephanie Evergreen! Lunch with you is always a treat, and your work continues to be the bomb. (BTW, that nail polish you liked is called OPI Scores a Goal. You. Are. Welcome.) And people, Ann Emery is taking the consulting world by storm.  Can I use the word pluck?  Look it up.  Pluck: spirited and determined courage. She’s completely inspiring.


  1. I had the good fortune to watch the Cartoon Guy (AKA Chris Lysy) in action from the front row of the Data Placemats session. Evaluation IS fun! (Sidebar: Chris challenged me last year to write a blog post about the conference – and I’m back for another round, so thanks for the kick-in-the-pants Chris!)


  1. At this point, you may be asking, “What is a Data Placemat?” It’s food imagery…and data…all in one conceptually aligned universe! What could be better than that?  Shout out to Veena Pankaj for her groovy presentation and for our sidebar conversation in the lobby lounge.  It’s always exceptional to learn more about what’s happening at Innovation Network.


  1. FullSizeRenderSpeaking of food and data theme, it’s a brilliant combination. Am I right? If you add a little vino and a bunch of evaluator author-types, you’d find yourself at the AEA Poster Session. (Which was AWESOME, by the way!)  Our poster channeled Flat Stanley by spending a bit of time in my hotel down the street, and then in Sheila Robinson’s room followed by Ann Emery’s digs, before I stopped by the handy-dandy FedEx office to ship it back home. Thanks to everyone who gave it shelter and stopped by to see it!  I look forward to hearing from you.


  1.   Continued my fangurling ways around Michael Quinn Patton. My goal this year is to actually reach out and introduce myself. (Consider yourself warned…I mean informed, MQP.)  I am so eager to dig into his new qualitative research book!


And the number 1 reason that I enjoyed attending the American Evaluation Conference:


  1. Sheila Robinson and Kim Firth Leonard just mentioned our blog in their session about writing effective survey questions.  WOOHOO!


It’s always a pleasure to spend time with people understand your discipline and get excited about professional growth.  I’d love to hear about your favorite professional development experiences, whether in person and online.  What conference, training or workshop would you recommend to a colleague?

The Cobbler’s Children…Well, You Know How It Goes

cobbler's children | Knowledge Advisory GroupYou’ve probably heard the phrase “the cobbler’s children have no shoes,” which is used to describe the phenomenon where professionals are so busy with work for their own customers or clients that they neglect using their skills to help themselves or those closest to them. I recently had an experience that reminded me of this adage and brought the concept home in a powerful way.

As a measurement professional, I’m often tasked with ensuring that my clients are asking themselves the right questions – tough questions – about their organizations, about their programs and services, and about themselves. I help them collect data so they can find answers, and then hopefully take action to support their goals. Measurement and program evaluation work are important in that way, as they give conscious attention to determining where we are and identifying the opportunities for improvement that every organization possesses.

Examining the core underpinnings and twisty edges of issues that are hard to grasp is a fundamental piece of my skill set. There’s nothing I enjoy more than watching the proverbial light bulb go off over the head of a client when I share a critical piece of data that we’ve uncovered or open their eyes to a new way of seeing things. And yet, when it comes to self-exploration, that’s a whole different thing altogether. I avoid it. It’s uncomfortable. It feels a little bit selfish. I’d rather spend time helping my clients than focusing on myself.

But avoiding our truths – be they strengths, challenges, or the way we engage with others – is never a good strategy. I believe this fully. I’ve seen many individuals and organizations blindsided, sometimes fatally, by a problem they could have easily tackled head-on if only they had been receptive to discovering and acknowledging it. And there’s no glory in denial when you’re running a nonprofit, government agency or business. Those issues that loom below the surface cost your organization every day in terms of staff energy, productivity, efficiency, and results. Even though you don’t see them, or perhaps choose not to see them, they still have an impact.

So, a few weeks ago, in the dead of winter and amidst a glorious snowstorm in Connecticut, I attended the Bregman Leadership Intensive. For three days, I had the opportunity – no, wait – I took the opportunity to immerse myself in a process of self-discovery. The Leadership Intensive was designed and led by Peter Bregman, who is the CEO of his own global management consulting firm ( as well as a thought leader in his field. Some of you know that I’m a big fan of Peter’s work, so I was excited to attend one of his sessions. Once there, I found myself surrounded by a small group of amazing people from around the globe who came there for the same reason. We wanted to discover something about ourselves, something about the way we approached our work and our lives, and find the courage to use that knowledge moving forward to be better leaders.

Gratefully, we had come to the right place. Instead of lecturing, Peter asked us those tough questions. He expertly guided, mentored, and supported us on how to tap into our own revelations, from within. Additionally, he and his team helped us inspect and recognize both attitudes and behaviors that we hadn’t seen before. Some were founded on unrealistic expectations and others were deep-seated fears, but all were eye opening.

Now, I’m not going to candy-coat this next part. This was not an easy journey for me. You may similarly find that looking at yourself and your organization, and especially being amenable to the notion that everything may not be perfect, is difficult. I had to be open to acknowledging realities and learning new things. I also needed encouragement from others who understood my challenges, possessed expertise, and pushed me to see new perspectives.

However, the benefits that I gained from the experience were invaluable. Here are just a few:
Clarity on what I want to accomplish,
• A sense of direction for how to get there,
Ideas for changes that may better enable me to reach those goals,
• And last but certainly not least, ongoing support from talented, incredible people.

In other words, I guess you could say that my own light bulb clicked on. How cool is that?

Introspection isn’t easy…but it’s certainly worth the effort. No matter how hard it may seem at first glance, it’s critical to stop, look, and listen to what’s happening around and within you. Take small steps, if needed. Be aware of the true realities, as well as the possibilities that emerge from learning and embracing new roadmaps. And above all, find the support that can help you make it happen.

You’ll find that you can turn on your light bulb too.

(By the way, if you’re not familiar with Peter’s writings or presentations, go to and jump on his email list. You won’t regret it.)

More Great Online Resources – March Web Tools

evaluations are scalable | Knowledge Advisory GroupThis month, we want to highlight some of the great online resources available to those in the nonprofit sector. These tools and sites offer free and low-cost options for bringing your nonprofit into the online world.

1. Impact Rising
This site is designed to help support social sector organizations and consultants by giving them access to the resources to engage in capacity-building projects. Here you’ll find things like organizational assessment tools, strategic planning resources & tools and fund development information.

It also seeks to make the norms and standards of the consulting industry explicit, so that social sector organizations know what to expect when partnering with a consulting organization.
Learn more at

2. Shoeboxed
This neat website allows paperwork to become paperless, streamlining your filing systems. With Shoeboxed, all your documents are scanned, categorized, and organized before they go into your online account, where you have access to them without shuffling through a filing cabinet.

All you do is sign up and send your documents in a Shoeboxed prepaid envelope, through their free uploaders, via email, or their free mobile app. They can even handle your important financial information – with IRS accepted receipt images – so that it’s ready to export to accounting or bookkeeping applications.
For more information, visit

3. TechSoup
TechSoup is a 501(c)(3) nonprofit that focuses on connecting social sector organizations with technology products and solutions. They also offer the learning resources needed to make informed decisions about technology. When an organization is registered with TechSoup, they can also access donated and discounted products and services such as high-quality refurbished hardware and software.
You can learn about joining TechSoup at

More Great Online Resources – Measurement & Surveys

HiRes-e1278130563745If you’re seeking great online resources, we’re here to help! Today, we’re highlighting Logic Depot, SurveyMonkey, TwtPoll, TwitSprout, and Statigram. These tools focus on measurement and surveys for your organization.

1. Logic Depot was developed locally, in Richmond, Virginia, and focuses primarily on providing clients with a set of self-service survey tools. The web-based software is free to organizations that collect fewer than 2,500 surveys a month and it has great functionality.  Logic Depot also offers project consulting, survey coding services, and custom integration. If you’re interested in learning more, let us know and I’ll connect to our friend Scott who can get you started!

2. is an easy-to-use, free online resource that allows anyone to make and administer surveys. You can use this tool to measure anything from participant feedback to program performance.

3. is similar to SurveyMonkey in that it enables you to create polls. You can share these polls on Twitter or other social networks. TwtPoll also allows you to see statistics related to your survey, such as when people voted or where they voted from.

4. If your organization is active on Twitter, then is a great, colorful way to track growth and activity on this social media network. TwitSprout makes it easy to convert your data into a shareable PDF format. It’s currently in a beta-testing mode, but it’s also offering free fifteen-day trials, as well as free and inexpensive options to track your activity.

5. Iconosquare is essentially the TwitSprout of Instagram. You can track Instagram statistics with Iconosquare, and the tool provides a variety of engagement data about your organization’s photos, likes, and follower growth. It also helps you discover the best and worst times to post on Instagram in order to help increase engagement.

What do you think of these measurement tools?

More Great Online Resources – Technology

iStock_000008376541XSmall11If you’re looking for great online resources for your business or nonprofit organization, look no further!

This month, we’re highlighting SimpleBooklet, Product Donations (by Microsoft), Screencast-o-Matic,, and npCloud. These resources help you create compelling stories, share information, and better serve your communities.

1. helps your organization create a web and mobile version of your company’s brochure, flyer, annual report, or other printed marketing materials. You can easily create online booklets that can be shared and promoted to multiple destinations on the web. Because this resource is hosted in the cloud, any tweaking or editing you do to a booklet automatically updates all versions wherever it is published.

2. Product Donations is a Microsoft program that provides free or discounted products for nonprofits. From cloud services to software, if you are a nonprofit in need of some technology tools, this is a great resource to check out.

3. Screencast-o-Matic is a popular screen recorder that simplifies the recording and sharing of screencasts. This free, web-based program allows organizations to easily create useful training for staff and volunteers, as well as motivating donors and advocates.

4. is a helpful tool that allows users to merge PowerPoint slideshows and video to create more persuasive multimedia presentations. As a nonprofit or a for-profit, you can use for awareness campaigns, volunteer and/or staff training, reporting, and creative storytelling. Because combines different presentation methods in new ways, it can be more compelling to viewers than ordinary presentation formats.

5. is a nonprofit service from Tech Impact. It provides cloud-based technology products and services (like data backup, calendars, servers and VoIP phone systems) to nonprofit organizations. npCloud allows you to use cloud-based technology in order to better serve your organization and the communities with whom you work.

Customer Surveys Part 4: Question Complexity

program evaluation terminology | Knowledge Advisory Group

Structuring questions appropriately is critical to collecting meaningful, interpretable information from customer surveys. In this last post of our blog series on creating effective customer feedback surveys, we will focus on tailoring the complexity of your survey questions.


The following questions demonstrate how alternative question wordings can allow you to target in on the specific information you want to gather.


How many times have you shopped in the Union Town Square during the past six months?


During the past six months, how many times have you shopped at the OhMySoap! store located at the Union Town Square?


Please think about any shopping visits that you made to the Union Town Square from June 1, 2013 through Dec. 1, 2013. During those visits, how many times did you purchase bath products at the OhMySoap! store?


The structure and wording for the last question is more complex that the others, but it creates a very focused request to the respondent, increasing the likelihood of obtaining truly useful information. As the structure of the questions changes to enhance clarity, you will be more likely to obtain specific, actionable information.


Are you asking customers to complete a survey in the near future? If so, will you be implementing any of our tips? We want to hear your stories!

Customer Surveys Part 3: Question Specificity


Question specificity is an important part of crafting your customer survey. Vague questions may not elicit useful information. On the other hand, overly specific questions may not be relevant to all of your survey respondents. Finding the right balance of specificity can impact the usefulness of the information you gather.

Continue reading

“Comprehensive data allowed management changes to be made more rapidly, leading to a leaner and more effective organization.” ~ American Red Cross, Richmond Chapter

“They were able to determine a ROI figure to help define the value proposition of our program.” ~ Virginia Economic Development Partnership

“The assessment allowed me to deliver a monumental, sea-change, amount of information to the organization with data to back me up.” ~ American Red Cross, Richmond Chapter

“Trina’s interviewing technique is a more effective way to uncover staff issues and ineffective procedures.” ~ Senior Connections, Area Agency on Aging

"The third party review added credibility.” ~ Virginia Economic Development Partnership

“The project more than paid for itself in the short-term.” ~ American Red Cross, Richmond Chapter

“I was spending 2-4 hours per week dealing with staff issues that have now been removed.” ~ Senior Connections, Area Agency on Aging

“We never received a report better than this one.” ~ Virginia Economic Development Partnership

"The Knowledge Advisory Group exceeded our expectations. They provided a solid grasp on what we achieved, along with additional framework for designing and measuring future programs."    ~Virginia Community Capital

Looking for a simple and affordable way to access our knowledge and expertise daily? Check out our newest program (spaces are limited!).