All Posts

Desktop reviews don’t have to be boring

  • Published: by Amy Samson | Permalink

Many evaluations I’ve worked on have either been focused around or include a desktop review. This could be of existing documents and/or trawling the web for other sources for information.

Trying to find relevant information is not always easy and can be very overwhelming, particularly when you have a big pile of documents to go through! It’s easy to fall into the trap of including too much information (just in case) into the report which then ends up huge and essentially just repeating what is already out there. This isn’t a very useful or analytical approach.

So I thought I’d share a few strategies which I have found helpful when having to trawl through a lot of information.

  1. Understand the end game. Is the review a main part of the evaluation or a contributing data source? What needs to come out of this report? An analytical conclusion or a summation of data?
  2. What are the key evaluation questions? Write them down. These guide your data collection and help keep you focussed on what information is important. Think of them as your interview questions and the data sources as your subject.
  3. Develop a report structure. This I find is the most important – it is your roadmap. By working out report headings (focused around the evaluation objectives) you’ll have a clear understanding of what information you need which means you won’t miss things and you won’t get lost. You should be able to intelligently read through your sources and address your headings. No one wants to have to go back over reports again and again re-checking data. But remember too that new information may emerge so you might need to add new headings as you go along – don’t become so focussed that you miss the little unintended outcome gems.
  4. Interrogate methodically. Go through your data sources methodically, one at a time, using your headings as a guide to pulling out the relevant information for your report. Don’t forget to attribute quotes and information (I find footnotes very useful for this) and include a list of all the data sources. Whether you state this up front or include in an appendix is up to you.
  5. Data presentation. Try to be as succinct as possible, use your own words and don’t data dump. By this I mean just cutting and pasting big sections of text from other reports etc. This isn’t useful and most likely, your audience has already seen this text many times and doesn’t need to see it again in your report! There may be different ways you can present the data. Could a table or graph be useful? Maybe an infographic could tell a great story?

While not exhaustive, hopefully these ideas give a little food for thought, help you to craft an interesting story and enjoy what is often seen as a hard slog!

Is doing some evaluation better than nothing?

  • Published: by Amy Samson | Permalink
Majorca Holiday

 

I was recently on holidays in Majorca (easy to do when living in London – not so easy from Australia!) where we stayed overlooking a lovely Mediterranean beach called Cala Esmerelda.

Now this is where I say something along the lines of while I was on holiday I was thinking about evaluation and had a ‘light bulb moment’ – except that didn’t happen at all! I was thinking about when I was going to have my next icecream, when the next swim was and did I want red or white wine with dinner.

That being said, the day we were leaving, a survey was left on the table for us to fill out about our experience at the resort. Being a sympathetic evaluator, I was compelled to fill it out, although I can imagine that many people intent on squeezing the last bit out of their holiday wouldn’t take the time. I wondered how many forms were filled out and whether the resort took action on the feedback they received? Was it worth their while?

Now I don’t know what other type of evaluation this resort does, but if this is it, at least they are doing something to find out what the end-user (us) thinks and hopefully acting on it. To me it demonstrates a frame of mind and it means that the organisation (hopefully) is thinking about improving.

In my opinion having some sort of evaluation is better than having none. Of course nothing beats a well-designed evaluation program feeding directly into organisational goals (and this may well be the case at the resort we visited), but being in the right headspace and seeking some sort of feedback is where it all starts.

Jeff has also updated his blog!

Winter 2015

  • Published: by Jeff Coutts | Permalink
Dutch CanalNorth Queensland Daintree North Queensland Sunset
The conference in Wageningen, The Netherlands, was great! It combined two extension streams – the Association of International Agricultural and Extension Education (US dominated) and the European Symposium on Extension and Education (European dominated) and brought about 380 delegates to the combined event. The theme was ‘Competence and Excellence in Extension and Education’ – and there was certainly a smorgasbord to choose from! I gave a paper on Changing Evaluation Approaches in Australia (see the papers section) and chaired a session on evaluation. It was fitting that the event was held in Wageningen – it has a long history of taking the lead in extension thinking and practice over the last decades. For me, it was where my family lived while I completed my Masters and then completed my PhD. Wageningen is also involved in the NZ Co-innovation project that is raising much World-wide interest in how to be innovative within an RD&E Stream.Of course, while over in Europe, I had chance to visit Amy and her husband Chris in London – and catch up with our grand-daughter Evie! [My wife, Robyn, was not able to travel with me on this occasion – so was very envious!]. It is great to have Amy being able to take a more active role in the company. Her leadership in the use and evaluation of social media is a timely addition to our team.

Given that it is winter, Robyn and I did get a break up in North Queensland in the middle of the year. This combined work with family visits – as well as a trip up to the Daintree. This really is the best time to visit the North! A very interesting stop was Paronella Park – the picturesque ruins of a Spanish ‘Castle’ built in the rain forest in the early 1900s. Quite an unexpected addition to the incredible rainforests and natural beauty in North Queensland!

With all the focus on agriculture with the White Paper and the R&D for Profit Program, I am expecting life to get even busier on the extension and evaluation front! I will keep you posted.

Keeping your ear to the ground – using social media as a listening post

  • Published: by Amy Samson | Permalink

As well as using social media to engage and build relationships with your audiences – it can be very useful as a listening post. While it might be interesting for project/program managers to understand what is being said online about relevant topics, this is probably more important for larger organisations and bodies who should be on top of what is being talked about online. Particularly those interested and impacted on by sensitive subjects.

To give you an example, in 2012/2013 (when a Senior Consultant with Gorkana) I worked with the Horizon Scanning and Futures team at the Department of Environment, Food and Rural Affairs (DEFRA) in England on their award winning social media horizon scanning program.

Among other things (including influencing policy decisions), this program looked to create an early warning system of emerging issues (e.g. environmental, wildlife etc) before the media gained interest. The aim was to pinpoint the conversations, understand who the influencers were and use this to inform the digital engagement strategy.

Using a social media scanning tool, and in constant consultation with the DEFRA team, we used keywords and phrases to monitor relevant issues and online conversations. This information was available live using online dashboards (allowing immediate action for evolving issues with daily/hourly reporting if necessary) and a written monthly report contributing to internal reporting to the highest levels of government. You can read an in-depth case study about it here.

Having some sort of social media listening post, even if it’s using free tools, is becoming more and more important. Organisations used to primarily monitor issues in the press using a clippings service (and they still do), but now the press is sourcing many of its stories from online
conversations.

You’re behind, potentially ambushed and in reactive mode already, if you don’t know the issues and how they are being talked about in social media.

This area is only growing in influence and policy makers are having to pay attention as social movements gain momentum ( change.org is a good example).

Should you be listening to social media?

Using Pinterest to collect and collate observational data

  • Published: by Amy Samson | Permalink

There are so many ways to collect data in today’s online world that simply weren’t available even 10 years ago. Information can now be collated in many different ways from even the remotest locations.

Observation is a data collection method that has been around for a while. One example of this methodology is the use of photos taken at regular intervals to record changes of land use over time. A particularly useful method for Natural Resource Management (NRM) projects focusing on land rehabilitation.

With digital photography, already it is easier to collect this data but sending large files via email can be restrictive (bouncing emails, inbox limits etc) and not everyone finds it easy to access online file sharing services.

And this is where Pinterest could be helpful. The site is all about sharing images, which makes it an ideal way to collate (using boards), sort and share photography.

Taking the above example of an NRM project, say a catchment area is being rehabilitated and Farmers participating in the project are asked to capture observations (with a set of guidelines) of change over time. The project officer could set up a Pinterest profile and create a series of boards which Farmers add photos to (including short written notes).

You would then be able to very easily see the changes over time in one space. It would also be compelling evidence for funders showing changes made and goals met.  And I’m sure those participating would enjoy seeing the results of their efforts so well represented.

The photos could be collated in a number of different ways depending on the reporting needs of the project including:

  • Monthly/seasonal boards – useful grouping change over time, helping comparison across Farms
  • A board per Farmer recording their observations
  • Boards focused around practices (e.g. tree planting)
  • Boards focused around plant types (e.g. grasses, shrubs)
  • Boards focused around benefits and outcomes

The same idea could be applied to programs focused around increasing production. Photos could be taken before, during and after the new/changed practice have been implemented. Ideally benefits and outcomes would be captured – for example decreased pesticide use or more available fodder for livestock.

Let your imagination go and be creative with data collection (of course while maintaining rigour). It doesn’t have to be that boring job that everyone procrastinates over.

Towards solving the social media evaluation mystery

  • Published: by Amy Samson | Permalink

In my experience social media as a communication channel can still feel overwhelming to many people. So taking the next step and thinking about how to meaningfully evaluate the use of it can almost feel too much of a mystery – which often means it doesn’t happen!

However, the more I’ve worked with social media evaluation the less mysterious I think it is. It’s easy to get caught up in the numbers that get thrown around like – we reached 20 million people with our tweet. To that I would say great, what does that number mean and so what?! (And there’s a whole other conversation to be had around how reach numbers are derived and what reach actually means.)

It’s no different to any other evaluation where you need to work out what success looks like for you. 

Understand where your social media activities fit into your overall objectives or Key Result Areas (KRAs) and build your measurements around this.  

Here’s a really quick example. You may have an overall KRA focused on encouraging on-farm practice change in one particular area. One strategy might be to use social media as a way to increase farmer and consultant engagement with the website and use the available information to inform practice change. Here are some ideas of what you could look at depending on your goals:

  1. Website traffic numbers – Do they increase when you include links in social media? Are people clicking through? Look at the traffic sources to see where people are coming from. 
  2. Quality of time spent on website – Are people spending longer on your site and are they spending it on the relevant pages? Has this time increased since you’ve been using social media?
  3. Engagement with content – Is the content on your site being shared through social media? Are your tweets being retweeted? are you being directly tweeted at and are you tweeting back? Are conversations happening on your Facebook page (if you have one). 
  4. Publication downloads – Are your publications being downloaded if you are promoting them? Are they being downloaded anyway? Who is downloading them and how are they being used (potential survey question)?
  5. Social media followers – Who are they? Are they the people you need to reach? Are more of the right people following you? Are you using the right channels? This could be something to ask in surveys/interviews. 
  6. Website awareness, use and contribution to practice change – Establish a baseline. Ask your target audiences (survey, interviews) whether they are aware of your website and how they currently use it. Ask again during and after your campaign. Find out if the information available contributed to any changes they’ve made, and are consultants are using it as a reference. TIP: Hijack other evaluations that might be going on and pop a couple of relevant questions in surveys, workshops, interviews etc.

In terms of understanding your social media strategy’s contribution to your KRA, point 6 is the most important particularly at the reporting stage. 

Every successful evaluation begins at the planning stage. This ensures that you collect meaningful data along the way. 

Plan to evaluate and you’ll plan to succeed. 

Autumn Update 2015

  • Published: by Jeff Coutts | Permalink
couttsjrblog_april2015_img1couttsjrblog_april2015_img2couttsjrblog_april2015_img3

It’s autumn and time for a blog update from me.  What is exciting for our web page is that Amy is also writing a blog and bringing in her insights and perspectives.  The topic of using social media in extension and its evaluation, for example, is an area that there is growing interest – not just in Australia – but world-wide.  She also has a twitter feed (@amyrsamson) – and I think I am going have to get more active in that area as well!  We are certainly using social media (Whatsapp) to get regular updates on Amy’s daughter Evie’s milestones – first crawl, first word (nana) and first standing up in the cot!  We are also using this app in some projects in which I am involved – with great effect.

In January we had our daughter Jocie (who we visited in Canada) home with our two grandchildren (via Disneyland in LA).  Visits to the Australia zoo and then searching for bearded dragons in our gully were hallmarks of the visit.  Who would have thought there was so much wildlife around (and we did find some!).

I have managed to get to the local Toowoomba Show. The show’s always been a highlight for me since my younger days showing cattle at the Royal Brisbane show when I was doing my undergraduate study. The highlight for me is definitely the sheep dog trials – my first port of call!

Going back to New Zealand to undertake a component of a project review in the dairy industry for Deloitte was a recent highlight.  I always love the trips to New Zealand.  I was reading the other day that at the time of Federation, there was a hope that New Zealand might have joined up at the time. Apparently there is “Manuka” in Canberra in anticipation of this happening!  Of course, then, the Cricket World Cup series final clash between Australia and New Zealand would not have happened!

The other NZ project in which I am involved – the Co-innovation Project – is also very exciting.  Not only does it consider the value chain but it seeks to link industries, public, private and farmers. It is about bringing the major players in the ‘biological industries’ to work together, learn from each other and make big jumps in a way that they might not have otherwise.  Australia is lacking here – it’s time to learn from our NZ counterparts and embrace ‘co-innovation’.  It’s a chance to find the big opportunities for change and progress.

Now I am taking a breath and having a short break with Robyn to visit temples in Cambodia and travel down the Mekong. A true holiday – with Ben and Amy holding the fort for the company.  And after that there is the conference in Wageningen, the Netherlands to participate in two international Extension conferences.  All designed to keep my mind active and to keep learning.  Extension is still vital for change – and its so nice to be a part of it!

Musing about the on-farm influence of social media

  • Published: by Amy Samson | Permalink

There are a few questions running around my head about social media and the rural space – particularly in relation to on-farm practice change.

What is its level of influence?

As a part of this I need to better understand how social media is being used by farmers, agencies, organisations, communities and government and for what purpose. And how is it being evaluated?

A recent ABC story on farmers using social media for their advantage talked about it being digital ‘word of mouth (WOM)’. As a communication channel, WOM has always been one of the most important and most trusted.

So is this trust translated online?

Social media is also being effectively used to build online communities. Take for example AgChatOZ which aims to raise the profile of Australian agriculture by shining a light on the leading issues that affect the industry and the wider community. If you’ve been on twitter at all and followed agricultural discussions, you will have come across its hashtag (#AgChatOZ) at one time or another. Again as an evaluator I would be really interested to see how its conversation has grown over time and what its impact has been.

I’ve also seen words such as empowering and engaging used in relation to social media and rural and remote Australia.

Its ability to connect the isolated and allow their voices to be heard is a big drawcard.

There’s a lot to talk about and I can see many blog topics ahead of me to address the questions that keep popping up and making thought bubbles over my head!

I’m really interested to know what is out there already so please do point me in the direction of any stories, studies, papers etc covering this or related topics. My email is: amy@couttsjr.com.au or find me on twitter: @amyrsamson

Back to basics and telling stories

  • Published: by Amy Samson | Permalink

TellYourStory
While writing this blog I thought how nice it was to feel the work cogs turning again and to read something other than parenting 101 blogs. Not recommended by the way, they have a frightening habit of making you feel like you’re doing it all wrong! In my short life as a mum (all nine months of it) I’ve learned that there is no perfect way to do anything but as long as you and baby are happy and thriving – you’re doing it right!

Anyway, to get my work groove on I thought I’d go back to the basics.

At its core, monitoring and evaluation as a practice is essentially about linking past, present and future actions through proper planning, goal setting, consistent data collection, analysis and effective reporting. And repeat.

If you follow this fairly straightforward formula, you have everything you need for robust story telling and feedback into the project, program or organisation.

In theory that is.

For the well over a decade I’ve worked in evaluation, I’ve felt that there has been a lack of focus on effective data collection. Which is crazy because this is the core of everything. We would see projects winding up, needing to conduct an evaluation and being unable to prove their impact properly because the right information hadn’t been collected in any sort of meaningful way. It was either not planned well, not carried out or collected in such a haphazard manner that making head or tail of it was almost impossible. Tough mudder for evaluators!

But there is change in the air.

What’s exciting is a distinct shift towards effective planning and implementation of data collection methods from the very start of a project (or continuously at the organisational level).

Action is being taken on a growing awareness that data needs to be robust and it needs to be targeted.

Collecting every possible piece of information is not only impractical but incredibly overwhelming when it comes to reporting on what matters.

Pressure is always increasing from funders, investors and regulators to prove that what you are doing and what they are throwing money at, is actually having an impact. They want to know if practices are changing, are natural resources are being managed better, has there been a return on investment etc etc etc.

As an evaluator it’s our job to tell your story, or to help you to tell yours in the best possible way. And I’m sure you’ll agree, that the richer and more targeted the data sources, the better.

Merry Christmas!

  • Published: by Jeff Coutts | Permalink
CouttsJRBlog_HopewellRocksCouttsJRBlog_WestBrisbaneAirportCouttsJRBlog_BuildingIvyCouttsJRBlog_CzechPaintingbyJeff
December finds us back in Canada for a family Christmas – the full complement of three children, spouses and three grandchildren! This time in New Brunswick where our daughter has family and has moved back to from their last posting in Quebec Province. From 30+degrees C in Toowoomba to -1 today and a nice covering of snow. The picture shows me at Hopewell Rocks – just outside of Moncton – remarkable because of the high tides and flower pot shaped rocks – and this time with a dusting of snow! It’s the seasonal contrasts that makes it all so exciting – between hemispheres and over the annual cycle.It’s been another full year with new projects and some new directions. From program reviews, project evaluations, visits to New Zealand and new M&E platforms for cotton, sugar and dairy. There has been on-going interest and work in developing M&E frameworks/guides for organisations and large programs. I enjoy this type of work because it establishes the basis for on-going rigorous M&E and provides a way for organisations and programs to better “tell their story”. There is remarkable work being done in the RD&E agricultural and natural resource management sectors – most of it under-reported and hence not understood.

This is also true with regards to the work being done to minimise negative effects of farming on the Great Barrier Reef. I was very pleased to see that coordination positions that were recommended in my review of Reef Plan Education and Extension have been put into place – and there is an even greater emphasis on evaluation of the impact of such strategies! This bodes well for the effectiveness of the combined government, industry, NGO and NRM body interventions seeking to assist with the change process occurring in farming.

A good development for me has been the opening of the “Brisbane West – Wellcamp airport” just half an hour from my house in Toowoomba! So far, Toowoomba – Sydney flights have started – and one that I have already taken advantage of. The picture shows the new airport at sunset when storm clouds were building over the Darling Downs. The house with the Ivy growing over it is one of the great sites around North Sydney where I stayed while doing some work for MLA.

Finally, I have continued to find time to dabble in acrylic painting. The painting attached was based on a photo that Robyn took of Český Krumlov in the Czech Republic! I am hoping to keep some space open next year for such creative outlets!

Thanks to all those who have worked with me over this year – and may you all have a good break before getting back into things early in 2015!