What or Who is Ufi learndirect?

“The concept of a ‘University for Industry’ led to the creation of Ufi in 1998. The organisation then set-up learndirect, a nationally recognised brand for learning. In six years learndirect has become the largest e-learning network of its kind in the world, and has individualised the delivery of learning to a mass audience through a unique combination of flexibility, accessibility and support.”

In this piece I plan to talk a bit about our e-learning platform and the part that open source tools and systems have played in our success.

Technical and Service Context

VOLUMES

447,000 learners last year

4,000 concurrent learners at peak

consuming 70 mb/s of bandwidth

99.98% systems availability

The learndirect learner management system (LMS) like most learning management systems is more than a website with lots of content.

Content sites like the BBC or CNN while they have some personalisation, typically present their consumers with a collection of web pages.If they are personalised at all they present their consumers with a sub-set of content according to preferences or tracked activity. Critically, the content itself does not change from consumer to consumer and as a result can be load-balanced across a number of serves or caches and requires relatively little tracking.

Learner management systems such as the learndirect system track a learner’s progress through a piece of learning and adapt in response to on-programme formative assessment. Such systems do expect to modify content according to consumer behaviour and as a result the use of multiple content servers only works to an extent. Such systems require a single authoritative data source for each course.

Additionally, consumers visiting a news or similar site have plenty of choice. If the BBC site is slow or not there for whatever reason, there are plenty of other such sites for a consumer to visit.

With web delivered learning, the consumer is intending to engage in a formal learning activity that they have formally enrolled in and in many cases have travelled to one of our learning centres to take their course. There is no other site for them to go to. If the site is slow or closed, then their journey was a waste of time.

For this reason the system must be both available and perform well. It is not enough that a system is available and returns content. If e-learning is to be effective, the medium needs to be as un-intrusive as possible; content has to render without the consumer becoming aware of any wait.

This presents us with a double bind; each user’s content is customised and there is a service expectation of 100% availability and responsiveness. In addition, we have issues of large scale and 24 x 7 availability we can see that constructing such a service is a serious web engineering exercise.

If you are not monitoring the service, then you are just running software.

It’s never good when the first person to tell you that your service has a problem is one of your consumers. Without appropriate monitoring software this will inevitably be the case, and in all probability they won’t tell you immediately.

So, the first key differentiator between a service and a system is Monitoring.

Choose the right tools.

When our service was first constructed a very expensive piece of software was purchased to perform availability monitoring, however, Mr. Heisenberg was forgotten and the load associated with that particular tool was sufficient to detrimentally impact the system. The tool itself was sold as the usual universal panacea, however, in implementation it was clear that its forte was component monitoring and not service monitoring.

Running a live system with this tool gave us all sorts of problems. The tool required agents on all machines and was really only designed around component availability and even then this was often measured from the wrong place (inside the firewall).

We took a look at the open source offerings available at that time and selected two.

Event monitoring

Nagios has won lots of awards. We use it to monitor events from two locations.

1. Our DMZ where it looks at all of our components every 90 seconds and critically has thresholds set for Green, Amber and Red. While most components in our large system are duplicated to provide resilience, it’s absolutely vital to know when one of your resilient components has failed in order to prevent a systems failure.
2. The public Internet. From this location, we can look at the service(s) from the perspective of the end user.

Nagios is used to provide event monitoring. Implementing such a tool is not to be undertaken lightly. Getting the sensitivity correct so as not to cry wolf, and embedding the culture such that when an alert is sent out, the operational staff respond rapidly is, in my opinion, more difficult than installing the system in the first place.

Trend and volume monitoring

The second open source monitoring tool we use provides trend monitoring, After looking around we found Cacti.

While Nagios tells us when we have a specific issue/problem, Cacti provides us with the information to understand or diagnose the root cause. In measuring volumes and their trends, Cacti allows us to look across the whole application stack at any point in time and examine critical volumes.

Cacti is used to measure volumes. If a system can return a number, Cacti can capture, store and trend it. These volumes can be business or technical volumes examples of which might include the number of users logged into the system over time or critical system volumes such as bandwidth, disk space, CPU, or Memory usage.

When you want to compare historical volumes or activity at a particular moment in time, Cacti can provide it.

Culture and tools

As you might expect from an open source tool set, both of these tools are highly extensible. We have been able to write and adapt agents to interface with them, with the exception of our database monitoring, and we have been able to monitor and trend all our services.

I spoke above about getting the culture right, putting these critical volumes onto big flat screens, making them obvious to everyone in your operations and service team. This was the single most important cultural change we made next to implementing an ITIL service culture.

The real question here is how we’ve been allowed to put all this instrumentation all over our application. Most government contracts are outsourced, but we chose to in-source our operations and development teams.

In-source, out-source, open-source, right-source

It’s about your technology strategy?

To understand this we need to talk about technology in a business context. Most organisations have either an implicit or explicit technology strategy. Within our organisation our Technology Strategy provides us with a framework that allows an organisation to make ‘good’, strategic choices, i.e. Hardware, software, monitoring systems, hosting providers. These choices are deployed within a governance framework to ensure that the business and service models that are dependant on technology can be delivered now and the future.

At the risk of stating the obvious, the selection of technology and service model an organisation chooses can mean the difference between a successful business and one that fails. As a consequence, organisations and IT directors tend to be conservative in there decision making.

At a simplistic level, technology is used for three things within an organisation:

1. to run the business
2. to change the business
3. to innovate

Unless you are a start-up, the bulk of investment and cost is already sunk in running your company. Changing the company IS usually occurs incrementally and takes the form of modifying the status quo. We are left with the shinny innovation tip of the cost iceberg to introduce new ways of doing things.

If we accept some of the above, we can see that technology strategies have considerable inertia, and unless there are some strong external pressures (failure to meet Service levels, company financial pressure, loss of market share), the adoption of new technologies is going to be slow. There is still a lot of COBOL out there!

So if you already don’t have a lot of open source in use, introducing it requires overcoming quite a lot of inertia.

As a company we have mandated the use of specific open-source operating systems and applications within our technology strategy where we can see cost and risk reduction. It’s worth saying that if our service was totally outsourced then these would not be our choices to make, other than at contracting and its very dangerous form to tell a supplier both what you want and how to do it.

In-source, out-source, right-source

The last ten years has seen the trend to out-source IT services and development continue to increase. This should not be a surprise when we consider the risk and cost of getting it wrong. Out-source companies come with the allure of having solved all problems previously and having a large pool of experienced staff and many organisations have significantly reduced the cost and risk of running their IT systems as a result.

Central to a successful out-source contract is a contract and a service description and underpinning set of requirements that are well defined. Good example candidates for out-sourcing are Payroll or Desktop management. In both cases, an organisation can describe what it is that it wants and the amount of change required going into the future can be estimated accurately.

It’s in the nature of our-sourced contracts that you describe to the supplier what you want but refrain from telling them how to do it.

If your IT application is the core of what your organisation does (such as the learndirect LMS) and you know you are going to undergo an annual cycle of change then in-sourcing your operations should be considered.

Having in-sourced the learndirect operations, we have seen a significant reduction in cost and have increased our service availability to > 99.9%

Open-source

If you have in-sourced your application development or hosting then you have the opportunity to exploit open source tools and applications for competitive or service advantage (are they the same thing?)

Having in-sourced the operations and now the development of our core application, we have put open source technology at the core of our technology strategy.

While we retain Oracle as our database of choice we have adopted a wide range of open source tools, Apache, SQUID, JBOSS, Hybernate, MySQL, Linux, to name but a few.

The advantages are obvious:

• They are standards compliant, or effectively comprise a cross-platform standard in their own right.
• They are robust and open to peer review such that issues and problems are rapidly identified and resolved
• They are often designed and built by practitioners and as such have solutions for real world problems built into them
• They increasingly come with support contracts

Summary

Looking back on what I have written it’s a bit rambling, however the key points I want to make are.

1. Don’t confuse running a Service and running an application. Monitoring and non-functional requirements such as usability, supportability, maintainability, availability make the difference.
2. Monitoring and its application is critical in running a service
3. Getting a technology strategy that supports the business and recognizes that once started it’s often expensive to change.
4. In-sourcing /out-sourcing right-sourcing will impact what you have control of.
5. Open source tools can be used to run world class infrastructure.

I hope you found something to make you think in this piece. We live in amazing times. The richest person in the world 10 years ago did not have one tenth of the knowledge we now have at our fingertips. Lastly, in the words of my favourite bumper sticker of all time, if you think that education is expensive try ignorance.

I want to welcome Dick Moore and thank him for agreeing to contribute to the Impact of Open Source Software and Open Educational Resources on Education series on Terra Incognita. His post is scheduled to appear on July 25, 2007 (eastern U.S.). Dick will share his experiences during the past 5 years at learndirect, during which time they have strategically adopted a range of open source tools and platforms that have helped transform Service and reduce operational costs. His posting will look at some of those tools, decisions and their service impact.

Dick Moore serves as Director of Technology at Ufi, where he looks after four teams that design, build and maintain learndirect’s IT infrastructure. The concept of a ‘University for Industry’ led to the creation of Ufi, which in turn serves as an umbrella organization supporting learndirect. Learndirect is the world’s largest publicly funded e-learning platform with in excess of 2,5 million learners.

I am very much looking forward to Dick’s posting. His will help tie together some of the key values of OSS that have emerged in the Series during the past few months. Special thanks to Seb Schmoller for recommending Dick and making the introductions. Please feel free to comment, ask questions, build on the conversation, and enjoy.

Dick’s posting can be found here http://blog.worldcampus.psu.edu/index.php/2007/07/25/running-a-service-not-a-system/

“Open Source Software and the User Experience in Higher Education,” the tenth installment of the Impact of Open Source Software Series, was posted on July 11th, 2007, by Mara Hancock who is with the Educational Technology Services at UC Berkeley. Thanks Mara for a great posting!

In her posting, Mara uses her direct experience with some community source projects and involvement with the Fluid project. She starts off my discussing the nature of usability and user experience, and makes clear that usability is not an issue exclusive to OSS, but OSS presents some fantastic opportunities and some significant challenges. The remainder of Mara’s post addresses some of these challenges. The challenges raised (and opportunities) of OSS as they relate to user experience and usability included:

• Distributed Teams: Although it is one of the powerful attributes of OSS, it also has the tendency to result in fragmentation of requirements based on local needs, and the creation of development silos.
• Code-Centric Culture: The currency of value in OSS is code and many usability professionals do not write code.
• Right People for the Right Job: User Interface and pedagogical expertise is not frequently hired into development teams.
• Flexibility of OSS: The flexibility that Open Code provides allows for incremental improvement based on local conditions, but that flexibility can also result in poor and inconstant user interface making testing very challenging.
• User-Friendly Architectures and Technologies: It is critical that an OSS application is friendly to the end user, but it must also be friendly to designers, developments, administrators, and other stakeholders.

To varying degrees, the Fluid project is addressing these challenges. In addition to providing a very engaging post, Mara also provided us with a number of useful links.

Comments
The comments concentrated principally on Mara’s insights around the relationships between software developers and lend users including learning designers and teachers. Open Source provides opportunities for better design for usability, but managers have to take advantage of the opportunities by hiring appropriate professionals and then providing time to actually work on usability. Additional questions were raised about the characteristics of open source communities that might produced better user experiences based on user engagement in the community. Finally, an observation was made about how the opportunities OSS offers for customization, and the desire for localization among many user groups, challenges usability testing.

Thanks again to Mara, for her engaging post and excellent responses to all questions, and to Michael Feldstein and other folks who have been reading along. Our next posting will be by Dick Moore, who serves as the Director of Technology at Ufi, on July 25, 2007. I am very much looking forward to Dick’s post. The schedule for the series can be found on WikiEducator.

Open source software has moved up the technology stack. We are now seeing consumer software such as Content Management, Learning Management, Portals, and other Web 2.0 tools all emerging directly out of open and community source efforts which provide unique opportunities for higher education to address the unique needs of their academic constituencies. But do we have what it takes to do this successfully? Do we have the right skills in our development shops? Can we bridge the divide created by distributed development teams to make for meaningful and seamless applications that will meet the work flows of all our users? How does the fact that we are in a teaching and learning environment impact that work and the methods we apply? This blog entry may be destined to ask more questions than I can answer! I hope that at the very least it might help to instigate a healthy dialog, illicit some emerging best practices from open and community source communities and from our — often less visible — local environments.

Let me enter into this conversation by way of a brief introduction to the work that has influenced my thinking in this area.

UC Berkeley has been actively working on Sakai since early 2005, when it was solely a grant and University funded project. We continue to be actively involved as it transitions to a full-fledged open source foundation model. I have been on the Sakai board of directors through this time. We are also a core member on the Fluid Project, recently funded by the Mellon Foundation, along with the University of Toronto (PI), Cambridge University, York University, UBC, and experts in usability, accessibility, and UI design across the globe. This community source project was created to focus on addressing the precarious value of UI and accessibility design in community and open source development work. In addition, UC Berkeley will be a core partner on the upcoming Kuali Student Project, which, as a project, has boldly declared a determination to be user-centered from day one. As you can see, as an organization UC Berkeley is deeply embedded in – and increasingly reliant on – open source applications, and in particular the community source projects, to deliver critical and integral functionality to our student and instructors every day. If these users – the heart and soul of our university’s endeavors – cannot use these tools to successfully fulfill their goals we are not doing our jobs.

Most of the findings in this entry are from personal reflection from my experience with the above community source projects, talking with colleagues involved in a variety of open source projects, and blogs and writing from across the web.

Delightful Software in Community & Open Source Software

When I first heard my fellow Sakai board member, CIO Brad Wheeler from Indiana University, refer to “user delight” as a strategic goal, I was slightly uncomfortable. The term “usability” is so much more utilitarian and sets a nice solid, non-evocative baseline. Don’t get me wrong, I want the BEST user experience possible, but “delight?” So I ask you, why not “user delight?” In fact, shouldn’t usable software simply be the bottom line? If we are going to be in the software development business, shouldn’t we be aiming to, at the very least, satisfy, and even better, create an experience that is welcomed — even sought after! Wouldn’t that be success?! In fact, when I step beyond my prudishness and my fear of failure, I do agree with Brad. Community and open source communities, where higher ed IT shops are striving to create superior software “by academia and for academia” are ideally positioned (at least theoretically) to achieve user delight. However, in order to do this we need to carefully examine the skills and resources and sometimes-unusual alliances that may be required to be successful in achieving this goal.

To begin, let’s be clear, poor usability in software applications is not relegated solely to the domain of open source. Many a commercial product has been slotted for demise – often prior to launch — because of poor usability. Indeed, as evidenced in many a UI listserv, UI design faces challenges in communicating its value across the spectrum of workplaces (spend a day or so on the IXDA list to observe this). Clearly usability problems are not the sole reason for what is reportedly an over 70% failure rate of software projects. But I would hazard to guess that if you are willing to broadly define usability as “a useful and satisfied user experience (UX),” and not just solely issues related to interface design, that a large portion of these failures are likely to indeed be tracked back to usability. While many of the symptoms experienced by commercial and open source development teams are similar, I expect that the solutions applied will often, and necessarily, be different in order to accommodate the cultural and organizational differences between the environments, as reflected in Eric S. Raymond’s “The Cathedral and the Bazaar.”

I have attempted to outline some of the challenges to the development of a delightful user experience in OSS and Community Source products from the perspective of those projects coming out of higher education for higher education. Many of these issues are interlacing and multi-layered and I don’t expect to create an all encompassing list, but to at least capture a general survey of some of the salient points.

Distributed Development Teams — The Good, the Bad, and the Inevitable

One of the huge benefits of developing open source products is that development can happen anywhere — and hopefully it does! In order to enable these distributed development teams to deliver in a timely manner, it is often necessary to create frameworks that allow the creation an implementation of loosely coupled tools. From many perspectives, this is a good thing to do: organizationally it allows open source teams to work efficiently (eliminate the coordination costs), and architecturally it provides much greater flexibility.

However, distributed teams introduce several UX challenges. Requirements developed in the silo of a remote team tend to focus on the requirements and business rules as expressed in that environment. For example, UC Berkeley might tend toward defining the business rules for the Gradebook based upon our campus policies rather than doing the extra work required to generalize across a wide range of institutions and global cultures. This behavior makes good local sense since as institutions we are driven by enlightened self-interest and need to ensure that we meet the needs of our local users with our local resources. However, producing a tool that only creates interactions based on the primacy of UC Berkeley’s business rules often effectively lowers the ability for other schools to leverage the tools and increases the total cost of adoption.

Another UX challenge is that working in tool silos makes it difficult to create a coherent, “holistic” environment for the end-user. Many user goals are based on work flows that cut across tool sets. This has been an oft-cited usability problem within Sakai. Users don’t think within the same categories and silos as the development teams work.

A Code-Centric Culture

Open source software has historically been developed for and by developers. It is a meritocracy where individuals gain respect through their direct contributions to the end product. This creates an intrinsic reward system for the developers whereby respect and privileges are accorded to those who do things like “play well with others,” provide good feedback and assistance, but most importantly contribute good, solid, workable code.

UI Designers generally don’t produce code. UI Designer Rashmi Sinha talks about this issue in her blog,

“…The problem of currency: In any system people exchange goods and services using some type of currency. The currency could be any arbitrary thing - it could be fish, cows, or massages. In the open source world, it happens to be code. The problem is that usability professionals generally do not write code.”

While quite successful for projects such as Linux and Apache, this is problematic for end-user applications that are used by the faculty and students in higher ed to support their daily scholarly, teaching, and learning activities. Developers can no longer design for themselves; they have to design for users whose goals are nothing like their own (a good read on this is Alan Cooper’s book, “The Inmates are running the Asylum“). Developers need UI Designers and Instructional designers to help them translate instructional, scholarly goals into specifications and prototypes. However, in an environment where code is king, what rewards are available for individuals with these other critical skills to participate? Do we even have the right ecosystem in which for them to engage them in the first place?

The Right People for the Right Job

As IT managers, we are probably the first to advocate for the right tool for the right job. However, we continually seem to hire a relative monoculture of IT professionals, thinking that if we just add another programmer all our problems will be solved! After talking with many IT managers across higher ed, it appears that UI design (whether it be User Research, Interaction Design, Visual Design, or Information Architecture) is rarely a formal part of their cycle or designers a regular part of the team. If UI Designers are part of the team, they are often so sparse a resource as to absolutely ensure that they won’t have enough time to get engaged early enough or long enough. This means that the few teams that are able to contribute UI designers to an open source effort, have a hard time being impactful. This is made worse by the fact that designers are often embedded in distributed teams and not looking across the product, inhibiting a holistic user-centered approach.

This inevitably creates a gap between expectations and deliverables and creates a tension that is exacerbated by the lack of recognition for UI deliverables that arrive unaccompanied by code.

Another challenge in creating applications for academia is that many of the user goals are embedded in pedagogical methods that may be discipline specific or not expressed in a generalizable way. Instructional designers and faculty are rarely part of a development team. In the higher education community source environment we have an opportunity to remedy this. It may require reaching across local organizational divides to ensure that the user and instructional goals are adequately being met: Often, instructors don’t speak the language of technology, so the instructional designer can help translate, generalize, and communicate their needs. In turn, the instructional designer often doesn’t speak the language of the application programmer, and the UI designer can help translate and represent their needs within the design and work flow of the application for the developers. This diagram attempts to express the relationship between these different areas of expertise.

The transparency of open source projects in higher education helps development and instructional support teams engage faculty and students in the process of creating the online environment that they need. We are uniquely situated smack dab in the middle of our own usability lab. There are few commercial or open source environments that can count themselves as this lucky. One of the biggest barriers to implementing a user centered design process that I have heard from UI Designers working in the private sector is their inability to gain consistent access to their users. Let’s make the most of our opportunity!

Flexibility of OSS

One of the largest benefits of open source software can also be a sizable UX challenge. The ability to easily localize and change the code means that often development teams and users don’t have a common or consistent experience and it is difficult to conduct user testing. On the one hand, proprietary products with closed code won’t let us make the experience more meaningful to our users, but on the other, with open source we have the unique opportunity to make a mess of it! An instance of uPortal at UBC may be completely different from uPortal at Yale. So how do we conduct usability tests? This issue is something that the Fluid Project is exploring now as it prepares for its first round of “user experience walk-throughs” on Sakai, Moodle, and uPortal. They have designed a set of protocols that are already being utilized by other community source projects.

User-Friendly Architectures and Technologies

Users of Open Source software are not only the end-users, they are also the designers, administrators, and implementation teams (hence documentation is also a huge barrier in OSS). When designing open source applications or platforms, making the software usable for these users is also important. In the case if UI designers, choosing presentation layer frameworks which are compatible with standard mark-up languages is important. The Fluid Project is working across projects, attempting to identify user interactions that cross academic software and develop accessible open source UI components that will be mapped to design patterns (http://developer.yahoo.com/ypatterns/, http://designinginterfaces.com/). For site administrators and integration teams, documentation and set-up wizards will be key. Design patterns for these activities should be able to also be mapped to reusable components and documentation templates.

While these issues only represent a subset of the details surrounding the challenges to creating delightful academic software, I think they highlight some of the opportunity as well. I am optimistic that through the technical, UI, and advocacy work of the Fluid Project and participating community and open source projects, we will be able to impact change both institutionally and within the open source organizations. I expect and hope that through forums such as Terra Incognita there will be more occasions for those of us in IT management, those engaged in supporting the teaching and learning endeavor, UI design, and programming across our campuses to find ways to bridge the divides both organizationally and culturally, and to collaborate in creating user-delightful open source software. You can find links to a number of related articles on UI design and open source usability on my del.icio.us site, tagged as “OSUI” (Open Source UI). Happy reading!

I want to welcome Mara Hancock and thank her for agreeing to contribute to the Impact of Open Source Software and Open Educational Resources on Education series on Terra Incognita. Her post is scheduled to appear on July 11, 2007 (eastern U.S.), but might appear on the following day. Mara will ask some questions and share some insights about the changing nature of OSS in teaching and learning and relate it to its impact on user experience and teaching and learning methods.

Mara Hancock serves as Associate Director for Educational Technology Services at UC Berkeley, and oversees the Learning Systems Group (LSG). She manages an extremely talented team of educational technologists, software programmers and architects, User Experience Designers, and training and support folks. We work with UC Berkeley faculty, students, and staff, as well as other educational technology professionals around the world to develop, adopt, and support collaboration and learning systems to enhance the teaching and learning experience.

In addition, Mara currently serves on the board of directors for the Sakai Foundation, a community source project developing a collaboration and learning environment (CLE) for and by higher education.

I am very much looking forward to Mara’s posting. Her topic is critical to teachers and learners as well as software developers and designers. Please feel free to comment, ask questions, build on the conversation, and enjoy.

“UNESCO’s Activities in FOSS For Education, Past, Current and Future Activities,” the ninth installment of the Impact of Open Source Software Series, was posted on June 27th, 2007, by Jean-Claude Dauphin of UNESCO’s Information and Society Division. Thanks Jean-Claude!

Jean-Claude’s posting was composed of two major sections. The first was an outline of the impressive portfolio of Free and Open Source Software (FOSS) and Open Educational Resources (OER) related projects that UNESCO leads or supports. In addition to providing a little background on UNESCO and its interest in FOSS, Jean-Claude also highlighted projects and activities ranging from a FOSS portal, to support of and participation in OSS projects, and community development and dissemination activities.

The second section was an outline for a most impressive future project. Jean-Claude outlined a “FOSS for Education” project that will result in a FOSS infrastructure designed to meet the needs of a university operating in developing regions. He provides a needs analysis, vision, rationale, and a skeletal project outline. His treatment of this project highlights the significant opportunity and also the magnitude of the work to be done.

Comments
The following comments and responses primarily concentrated on clarifying needs, the role of UNESCO, and some of the challenges and dependencies for the success application of the project. Jean-Claude pointed out where FOSS can best be leveraged in education and government in developing countries and then dug into some of the issues around the economics and accessibility of online education and the role that FOSS can play in relieving constraints. I have an additional follow-up question about the role of customization to support local needs, which I will post soon.

Thanks again to Jean-Claude, for his visionary post and excellent responses to all questions, and to all of the other folks who have been reading along. Our next posting will be Mara Hancock, who serves as the Associate Director for Educational Technology Services at UC Berkeley, on or around July 11, 2007. The schedule for the series can be found on WikiEducator.