A Pedagogical Richter Scale for Earth-Shattering Experiences

Transformation Rubric for Engaged Learning: A Tool and Method for Measuring Life-Changing Experiences

Is an article authored by Emily Springfield and Anne Gwozdek from U of Michigan School of Dentistry and Andrew P. Smiler, of Evaluation and Education Services, LLC. It’s published in the latest edition of IJeP.

In brief, they argue that although higher ed is proficient at assessing competencies, it lacks the tools to asses “affective aspects of learning, such as changes in perspective and identity.” Their rubric is both a tool and a method of measuring “a program’s impact beyond competency attainment”.

The authors begin by describing the online, E-Learning, dental hygienist program that led to the problem the rubric attempts to solve: the need for a tool to measure the “personally transformative” learning from the traditional assessments of competencies (64).

The next section, “Need for a Rubric,” is an inspiring description of the program based on the powerful testimonies of its students that “Something special” is happening in it. Their curriculum asks student to compose self reflections often and the final project is to compose a meta-reflection on all the reflections they made throughout the program. The response to that method seems to be where the magic is happening based on their students’ testimonies:

Students say of the program “it changed my life” or “I see the world in an entirely new way now.” Even mature students—those in the E- Learning Program are coming back to college after an average of 7 years in professional practice—with personalities not generally given to exaggeration, report that “This is the best thing I’ve ever done,” and “I didn’t really understand at first but after the last round of reflections, I really started to get it why we are doing all these extra things.” (64)

This article is a powerful resource for gathering data on high impact learning/ teaching practices. The most useful and interesting content is where the researchers describe their coding process for developing the rubric. Although labor intensive (developing and applying the rubric), the value is immediately apparent as the researchers explain in detail their reasoning and convincingly argue for their assessment themes: Confidence, Pride, Skills, Perspective, and Identity, as well as the types of deliverables that might best be assessed: eportfolios, self reflective essays, exit interviews, open ended survey questions, online threaded discussion questions, etc.

Here’s an example of the questions used to gather the data from Table 1:
Focus Group Questions Used in the E-Learning Program
The focus group data used to develop the coding rubric had the following questions:
•    Did you have any a-ha moments?

•    What do you see as the role of reflection in your profession moving ahead?

•    Do you notice differences between yourself and the people you work with vis a vis reflection?

•    Can you identify something you do differently as a result of being reflective?

… (68)

There’s a lot more included in the article detailing the steps and process of implementation.

We should consider using this tool as we engage in this curricular change in FYC. We can use it to measure where students are now in the current system and how eportfolio practice affects student perception on learning. Implementation will take more than simply copying the survey questions, but I think designing a plan to implement it to gather data over the next 3 years would be a wonderfully worthwhile mechanism for assessing the value of the eportfolio in FYC.

Advertisements

“We Wouldn’t Call Agronomy ‘Shovel’ “: Trent Batson on renaming the eportfolio

Trent Batson, (aka Godfather of the eportfolio) has a blog on the AAEEBL website.

(AAEEBL (“able”), the Association for Authentic, Experiential and Evidence-Based Learning, serves as a professional association dedicated to supporting educational leaders committed to educational transformation relevant to 21st century learners.  We are best-known for promotion of eportfolios as a high impact practice that encourages learners to reflect on their learning and to learn more deeply as a result of their reflection process.)

A few days ago he posted an entry bemoaning the name “eportfolio” and arguing for a new acronym, PEBL:

Portfolio Evidence-Based Learning (PEBL)

He mentions many of the same complaints about the name (its multiplicity of meanings and the way the term limits understanding to a piece of technology rather than a process of learning)  as I did in my last post, and several more important ones I didn’t think about.

He also makes a case for his new proposed term: PEBL

One of his main concerns is that the new name reflect the concept of evidence based learning and stay true to its both-a-process-and-a-product roots:

“But, as we consider our nomenclature, we probably don’t want to eliminate “eportfolio.”  Using only “evidence-based learning” as our actual academic and research field, would cut away our eportfolio roots and make us seem just one more of dozens of x-based learning movements.  Our community, and AAEEBL, are unusual – maybe unique – in championing both an approach to learning and the technology that enables it.  This is why I think we might want to consider “Portfolio- Evidence-Based Learning (PEBL).”

He goes on to assert that the field known as “eportfolio” is an oddity in educational circles, but changing the name to PEBL will connect the community more solidly to a “tradition of constructivist learning varieties”. In his mind, the term Evidence is the lynchpin of the acronym. He makes two claims in assertion of this:

first, a much fuller, more detailed, multi-modal set of evidence of unmonitored work to link to disciplinary concepts, and, second, access to a personal collection of such evidence gathered over time to integrate.  Making connections to previous ideas and discoveries is at the core of learning.”

I don’t fully understand what he means by “unmonitored work to link to disciplinary concepts.” It seems to me this is a bit reductive, or at least oversimplfied, as it prioritizes the action of “Linking” rather than “Reflection.” In fact, although I completely support his call to re-name eportfolio and his reasoning, I’m not sure I’m sold on his proposal. Although “PEBL” does sound a lot better than “eportfolio”, I’m not convinced that “EVIDENCE” is the crucial thing to name.

After all of the research and examples we’ve looked at over the past couple months, I think it is REFLECTION that describes the value of this pedagogical (constructivist) approach. To me, his arguments for the term “Evidence” beg the question of how/why the term “Portfolio” is necessary as the leading letter of the acronym. What is the distinction to be made, in other words, between “Portfolio Evidence”? Portfolio, to me, is an intentional collection of evidence (i.e. artifacts). In other words, the “PE” in “PEBL” is where the term fails in my mind to convey what Batson describes as a crucial distinction.

I understand why the word “Portfolio” itself would be useful in that it would like the new term to the old, but it reads as an awkward appendage: Portfolio Evidence Based Learning. Either we read it as an adjective (although, again, this causes confusion because the Evidence is not “portfolio evidence” but rather “learning evidence” that is collected in a portfolio) or as a double noun (Portfolio & Evidence based learning does not make sound linguistic or pedagogical sense either).

In the last paragraph we get a final argument and here I think is where more conversation about the nomenclature needs to happen: “PEBL is a theory itself – evidence used on behalf of developing the metacognitive aspects of reflection and integration – that also adds enormous dimensionality to existing learning theories and practices.”

PEBL is a better name than eportfolio, but it does not denote its own theory in an intuitive way. To be clear, I think the real failing of eportfolio as a term is the ugliness of the word–that overly-trite digital portmanteau makes it seem dated and corporate–and the fact that it does not describe its own value. (For example, every time I talk about eportfolio integration in FYC and our research project, people shrug and say “Well, just have them create an eportfolio in first year composition.” The problem is not a misconception, but an oversimplification that the term itself causes.)

I think a new term is needed but I’m not sold on PEBL. It fails in the same way the old term does. I think a term that is of use to educators must incorporate one/all of the following terms that Batson has to use to describe the value of the theory behind “PEBL”: “Meta-cognitive” “Reflection” “Integration” other conceivable synonyms for conveying the key process at the heart of portfolio learning: “Constructed Learning” or “Meaning Making” or “Creative Combination”

The more I think about it, the more I return to the acronym I came across (and discussed in an earlier post: Thoughts on “Fostering Integrative Knowledge”) in a study done by the University of Michigan: KIPP (Knowledge Integration Portfolio Process). Every word in that acronym moves the definition forward and makes sense, without redundancy, of its value to all stakeholders. In other words, it does what Batson calls for and it sounds like a sounds like pedagogical practice. The only thing lost is that it is not as easily used as a noun in a sentence: “My students are working on their KIPPs.” Maybe we would say “…their KIPP projects”? I don’t know. What are your thoughts?

Do Employers Value ePortfolios?

SUNY article / blog post: Do Employers Value ePortfolios

References this study, a national survey of Business and Non Profit Leaders, that was solicited by the AACU: It Takes More than a Major: Employer Priorities for College Learning and Student Success: Overview and Key Findings

Which is a digested version of the actual report, of the same title, which was conducted in 2013 by Hart Research Associates

Of note for our purposes is finding number 8 specifically (sorry for the poor image quality):8

Questions

As I’ve been reflecting on our research over the past week or so, I have some questions I want to share–perhaps as subjects for further research or just to discuss at our next meeting:

How many universities currently use eportfolios and on what scale?

How many of those use it as a tool for assessment (and use an internal platform to host students portfolios like emma at UGA for example) and how many use it as a professional development / optional tool (like Auburn does where even your platform is a matter of personal choice–Wix, Weebly, etc.)?

Which University’s approach is the best of the ones you’ve encountered so far? My vote is split between University of Michigan and University of Mississippi.

What research exists on their potential for helping students get jobs or get into grad school?

————————-

Shifting Gears: Questions about Stakeholder Buy In

What are going to be the likely objections faculty have? Will TT, NonTT and GTAs have different concerns / objections / anxieties?

Is their resistance a product of ignorance (And how do we overcome that effectively)  or resistance to the practice of integrative learning–i.e. they like their silos too much?

As far as students go–is it possible to make a student understand the value of integrative learning at the age of 18, 19, 20, 21, etc.? It kind of feels to me like the value of eportfolio demands wisdom, self-regulation, and a broad view of life’s challenges that most of my freshman lack. For that reason it kind of feels like the eportfolio might be an “If you build it they will come” kind of project. I anticipate resistance summed up, for example, by this anonymous student’s gripe on the Talk Page of the Eportfolio entry in Wikipedia:

This portfolio is pointless. I feel that we the students dont need to show what we learned 3 times every year. We all ready do the mid-terms and the final exams, so why should we have to make a profolio showing the work we did in class. It’s like makeing a folder to show your parents what you learned in school today. I for one think this is possably the dumbest idea ever thought up, its right up there with the war in Iraq and the Articles of Confederation. —The preceding unsigned comment was added by 158.123.138.50 (talk) 15:04, 13 December 2006 (UTC).”

Spelling, grammar, and grossly inappropriate analogies aside, the complaint speaks to a real problem we need to anticipate and plan carefully to address in our presentation of the project to both our students and our colleagues.

Assessing the Future: Eportfolio Trends, Uses, and Options in Higher Education

https://net.educause.edu/ir/library/pdf/ERB0904.pdf

A brief, nuts and bolts synthesis of the state of ePortfolios in higher ed by two administrators from Johns Hopkins. This article does a great job of categorizing stakeholders, uses, and best practices in the implementation of eportfolios as well as some interesting reasons why not to adopt them on a campus: “The do-nothing decision” is always a viable option (9). The report ends with a list of key questions to ask for anyone considering the adoption of the tool. This is a great source to look at as we consider best practices and potential arguments from the various stakeholders likely to be resistant to our proposal to implement eportfolios on a curricular level which  (I hope I’m not misrepresenting your feelings about this, Scott) seems like the direction our research is pushing us into. Assessing the Future: E-Portfolios

Do Employers Really Care About ePortfolios? (i.e. Oh, what’s in a name?)

http://www.educause.edu/ero/article/e-portfolios-hiring-tool-do-employers-really-care

This brief article presents results from a survey of just over 5, 000 employers with a response rate of 13%–so just under 700 employers–on whether or not they do or would consider using eportfolios to make hiring decisions.

Here are the results (copied from the article):

Results of the Employer Survey

Use of e-portfolios by HR managers was low for all majors, but the data revealed higher use in the fields of educational services and health care/social assistance. The reasons companies gave for not using e-portfolios were:

  • Not familiar with e-portfolios (75 percent)
  • Not valuable (14 percent)
  • Time constraints (13 percent)
  • Cost (12 percent)

Of those companies that did use e-portfolios, 16 percent used them in the initial screening stage, and a small percentage (

The type of information employers believed would be valuable in an e-portfolio included the following:

  • Resumes/references (93 percent)
  • Written work (39 percent)
  • Projects (37 percent)
  • Presentations (33 percent)
  • Lesson plans (23 percent)
  • Case studies (7 percent)
  • Artistic performances (6 percent)—————- End of the results————-
  • My question after reading this is, have there been other studies conducted on this issue? I haven’t searched beyond the first few Google hits, but this article seems to be the one most often cited so I’m thinking perhaps it is the only one?
  • A Rose By Any Other Name…
    Also, has anyone looked at whether or not calling it by a different name  (i.e. multi-modal resume, or interactive resume, or dynamic employee profile, or just eResume, etc.) in this context of job searching–of appealing to the specific audience of potential employers makes a difference?
  • It seems the biggest problem according to the survey results (and common sense) is that the name is off-putting and non-intuitive. Unless the specific employer is familiar with the concept (not the name itself but the actual practice and process that the tool is supposed to encourage) then it’s likely their opinions about its usefulness are going to be skewed by nonrelevant factors.

Active Classroom and Cyber Spaces

A copy of my blog post from today:

The excellent website on eportfolios maintained by the Rhetoric and Writing Department at the University of Mississippi contends that since eportfolio pedagogy consistently involves the students directly in their own education, it is then, in fact, active learning or, more precisely in their own words, “student-centered” learning.  This latter phrase, of course, is what leads some teachers to reject the idea.  In addressing this particular concern, the website authors say that “[t]hough ‘student-centered learning’ has become a cliché [,] such over-generalizations draw their power from an understanding that our students need to be more involved in deciding what is studied, how it is studied, and what learning is significant in relationship to class goals. This does not mean that we are surrendering our role as teachers. Rather, it acknowledges that students experience more meaningful learning and are more apt to keep and apply what they learn when they are invited to help select, in a dialogue with their teachers [my emphasis], what activities they will engage in to improve their understanding of course content.” I would also add that by involving them more in their own education, we are improving the likelihood that they will recognize and acknowledge more of their own responsibility.

What I hope to do this fall is to merge the benefits of active learning exercises and collobarative work in the physical classroom with the cyberspace equivalent through the eportfolio experience in such a way that when they leave the classroom they still carry it with them in their heads and on the internet.  They will continue to observe, think, and reflect periodically as they go through their other activities and then share the outcomes with their peers in the classroom and through the web.