Desktop reviews don’t have to be boring
Many evaluations I’ve worked on have either been focused around or include a desktop review. This could be of existing documents and/or trawling the web for other sources for information.
Trying to find relevant information is not always easy and can be very overwhelming, particularly when you have a big pile of documents to go through! It’s easy to fall into the trap of including too much information (just in case) into the report which then ends up huge and essentially just repeating what is already out there. This isn’t a very useful or analytical approach.
So I thought I’d share a few strategies which I have found helpful when having to trawl through a lot of information.
- Understand the end game. Is the review a main part of the evaluation or a contributing data source? What needs to come out of this report? An analytical conclusion or a summation of data?
- What are the key evaluation questions? Write them down. These guide your data collection and help keep you focussed on what information is important. Think of them as your interview questions and the data sources as your subject.
- Develop a report structure. This I find is the most important – it is your roadmap. By working out report headings (focused around the evaluation objectives) you’ll have a clear understanding of what information you need which means you won’t miss things and you won’t get lost. You should be able to intelligently read through your sources and address your headings. No one wants to have to go back over reports again and again re-checking data. But remember too that new information may emerge so you might need to add new headings as you go along – don’t become so focussed that you miss the little unintended outcome gems.
- Interrogate methodically. Go through your data sources methodically, one at a time, using your headings as a guide to pulling out the relevant information for your report. Don’t forget to attribute quotes and information (I find footnotes very useful for this) and include a list of all the data sources. Whether you state this up front or include in an appendix is up to you.
- Data presentation. Try to be as succinct as possible, use your own words and don’t data dump. By this I mean just cutting and pasting big sections of text from other reports etc. This isn’t useful and most likely, your audience has already seen this text many times and doesn’t need to see it again in your report! There may be different ways you can present the data. Could a table or graph be useful? Maybe an infographic could tell a great story?
While not exhaustive, hopefully these ideas give a little food for thought, help you to craft an interesting story and enjoy what is often seen as a hard slog!
Is doing some evaluation better than nothing?
I was recently on holidays in Majorca (easy to do when living in London – not so easy from Australia!) where we stayed overlooking a lovely Mediterranean beach called Cala Esmerelda.
Now this is where I say something along the lines of while I was on holiday I was thinking about evaluation and had a ‘light bulb moment’ – except that didn’t happen at all! I was thinking about when I was going to have my next icecream, when the next swim was and did I want red or white wine with dinner.
That being said, the day we were leaving, a survey was left on the table for us to fill out about our experience at the resort. Being a sympathetic evaluator, I was compelled to fill it out, although I can imagine that many people intent on squeezing the last bit out of their holiday wouldn’t take the time. I wondered how many forms were filled out and whether the resort took action on the feedback they received? Was it worth their while?
Now I don’t know what other type of evaluation this resort does, but if this is it, at least they are doing something to find out what the end-user (us) thinks and hopefully acting on it. To me it demonstrates a frame of mind and it means that the organisation (hopefully) is thinking about improving.
In my opinion having some sort of evaluation is better than having none. Of course nothing beats a well-designed evaluation program feeding directly into organisational goals (and this may well be the case at the resort we visited), but being in the right headspace and seeking some sort of feedback is where it all starts.
Jeff has also updated his blog!
Keeping your ear to the ground – using social media as a listening post
As well as using social media to engage and build relationships with your audiences – it can be very useful as a listening post. While it might be interesting for project/program managers to understand what is being said online about relevant topics, this is probably more important for larger organisations and bodies who should be on top of what is being talked about online. Particularly those interested and impacted on by sensitive subjects.
To give you an example, in 2012/2013 (when a Senior Consultant with Gorkana) I worked with the Horizon Scanning and Futures team at the Department of Environment, Food and Rural Affairs (DEFRA) in England on their award winning social media horizon scanning program.
Among other things (including influencing policy decisions), this program looked to create an early warning system of emerging issues (e.g. environmental, wildlife etc) before the media gained interest. The aim was to pinpoint the conversations, understand who the influencers were and use this to inform the digital engagement strategy.
Using a social media scanning tool, and in constant consultation with the DEFRA team, we used keywords and phrases to monitor relevant issues and online conversations. This information was available live using online dashboards (allowing immediate action for evolving issues with daily/hourly reporting if necessary) and a written monthly report contributing to internal reporting to the highest levels of government. You can read an in-depth case study about it here.
Having some sort of social media listening post, even if it’s using free tools, is becoming more and more important. Organisations used to primarily monitor issues in the press using a clippings service (and they still do), but now the press is sourcing many of its stories from online
You’re behind, potentially ambushed and in reactive mode already, if you don’t know the issues and how they are being talked about in social media.
This area is only growing in influence and policy makers are having to pay attention as social movements gain momentum ( change.org is a good example).
Should you be listening to social media?
Using Pinterest to collect and collate observational data
There are so many ways to collect data in today’s online world that simply weren’t available even 10 years ago. Information can now be collated in many different ways from even the remotest locations.
Observation is a data collection method that has been around for a while. One example of this methodology is the use of photos taken at regular intervals to record changes of land use over time. A particularly useful method for Natural Resource Management (NRM) projects focusing on land rehabilitation.
With digital photography, already it is easier to collect this data but sending large files via email can be restrictive (bouncing emails, inbox limits etc) and not everyone finds it easy to access online file sharing services.
And this is where Pinterest could be helpful. The site is all about sharing images, which makes it an ideal way to collate (using boards), sort and share photography.
Taking the above example of an NRM project, say a catchment area is being rehabilitated and Farmers participating in the project are asked to capture observations (with a set of guidelines) of change over time. The project officer could set up a Pinterest profile and create a series of boards which Farmers add photos to (including short written notes).
You would then be able to very easily see the changes over time in one space. It would also be compelling evidence for funders showing changes made and goals met. And I’m sure those participating would enjoy seeing the results of their efforts so well represented.
The photos could be collated in a number of different ways depending on the reporting needs of the project including:
- Monthly/seasonal boards – useful grouping change over time, helping comparison across Farms
- A board per Farmer recording their observations
- Boards focused around practices (e.g. tree planting)
- Boards focused around plant types (e.g. grasses, shrubs)
- Boards focused around benefits and outcomes
The same idea could be applied to programs focused around increasing production. Photos could be taken before, during and after the new/changed practice have been implemented. Ideally benefits and outcomes would be captured – for example decreased pesticide use or more available fodder for livestock.
Let your imagination go and be creative with data collection (of course while maintaining rigour). It doesn’t have to be that boring job that everyone procrastinates over.
Towards solving the social media evaluation mystery
In my experience social media as a communication channel can still feel overwhelming to many people. So taking the next step and thinking about how to meaningfully evaluate the use of it can almost feel too much of a mystery – which often means it doesn’t happen!
However, the more I’ve worked with social media evaluation the less mysterious I think it is. It’s easy to get caught up in the numbers that get thrown around like – we reached 20 million people with our tweet. To that I would say great, what does that number mean and so what?! (And there’s a whole other conversation to be had around how reach numbers are derived and what reach actually means.)
It’s no different to any other evaluation where you need to work out what success looks like for you.
Understand where your social media activities fit into your overall objectives or Key Result Areas (KRAs) and build your measurements around this.
Here’s a really quick example. You may have an overall KRA focused on encouraging on-farm practice change in one particular area. One strategy might be to use social media as a way to increase farmer and consultant engagement with the website and use the available information to inform practice change. Here are some ideas of what you could look at depending on your goals:
- Website traffic numbers – Do they increase when you include links in social media? Are people clicking through? Look at the traffic sources to see where people are coming from.
- Quality of time spent on website – Are people spending longer on your site and are they spending it on the relevant pages? Has this time increased since you’ve been using social media?
- Engagement with content – Is the content on your site being shared through social media? Are your tweets being retweeted? are you being directly tweeted at and are you tweeting back? Are conversations happening on your Facebook page (if you have one).
- Publication downloads – Are your publications being downloaded if you are promoting them? Are they being downloaded anyway? Who is downloading them and how are they being used (potential survey question)?
- Social media followers – Who are they? Are they the people you need to reach? Are more of the right people following you? Are you using the right channels? This could be something to ask in surveys/interviews.
- Website awareness, use and contribution to practice change – Establish a baseline. Ask your target audiences (survey, interviews) whether they are aware of your website and how they currently use it. Ask again during and after your campaign. Find out if the information available contributed to any changes they’ve made, and are consultants are using it as a reference. TIP: Hijack other evaluations that might be going on and pop a couple of relevant questions in surveys, workshops, interviews etc.
In terms of understanding your social media strategy’s contribution to your KRA, point 6 is the most important particularly at the reporting stage.
Every successful evaluation begins at the planning stage. This ensures that you collect meaningful data along the way.
Plan to evaluate and you’ll plan to succeed.
Musing about the on-farm influence of social media
There are a few questions running around my head about social media and the rural space – particularly in relation to on-farm practice change.
What is its level of influence?
As a part of this I need to better understand how social media is being used by farmers, agencies, organisations, communities and government and for what purpose. And how is it being evaluated?
A recent ABC story on farmers using social media for their advantage talked about it being digital ‘word of mouth (WOM)’. As a communication channel, WOM has always been one of the most important and most trusted.
So is this trust translated online?
Social media is also being effectively used to build online communities. Take for example AgChatOZ which aims to raise the profile of Australian agriculture by shining a light on the leading issues that affect the industry and the wider community. If you’ve been on twitter at all and followed agricultural discussions, you will have come across its hashtag (#AgChatOZ) at one time or another. Again as an evaluator I would be really interested to see how its conversation has grown over time and what its impact has been.
I’ve also seen words such as empowering and engaging used in relation to social media and rural and remote Australia.
Its ability to connect the isolated and allow their voices to be heard is a big drawcard.
There’s a lot to talk about and I can see many blog topics ahead of me to address the questions that keep popping up and making thought bubbles over my head!
I’m really interested to know what is out there already so please do point me in the direction of any stories, studies, papers etc covering this or related topics. My email is: email@example.com or find me on twitter: @amyrsamson