Personal apology to UWA regarding 2012 FSAE-A Design Event

[Big Bird]

Firstly, I am writing this of my own volition. I do not claim to have any knowledge of the opinions of any other people or organizations. These are my words, I will own them and I will stand by them.

In 2012, the UWA FSAE team presented a unique suspension concept on an incomplete car at the FSAE-A competition. The car was awarded 3 points in the Design Event. I was the Executive Director of SAE Australasia at the time, and had outright executive control of the event. The chain of command stopped with me.

The Design judging team was operating under my command at the time. I did not intervene or ask for explanation from the Design Event judging team as to why the team received such a low score. I was aware of precedents prior to 2012 where the Design Event judging team had used its better judgement to reassess a team's score when it appeared the score did not match the true design knowledge of the team.

Given that I had the design process knowledge and, at the time, nearly 12 years experience in Formula SAE, I had sufficient understanding to know that 3 points was not an adequate score. I had spoken to the team both at the event, and prior to that a number of times during the year. I was very impressed with their knowledge and their creativity. Therefore I knew that their final score was not representative of their design knowledge. However I did not take any action to have their score investigated.

As such I take full responsibility for this oversight, and I offer the UWA team my sincere apology. I recognize that the score given was hurtful and even insulting to a team that was willing to take the risk of being innovative. We should be nurturing such inventiveness, for the sake of our future industry.

Furthermore, I heard reports of officials addressing UWA team members in a manner not appropriate to a student design competition. I did observe instances of officials being unnecessarily aggressive in their discussions about the UWA entry, although i do not remember seeing specific instances of inappropriate conduct towards the UWA team members. If such inappropriate conduct occurred then I accept that it occurred under my command and I will gladly work with the UWA team to make amends.

There are some in this community who might see apology as a sign of weakness. I do not. I believe an injustice occurred under my command, and for that I offer the UWA team my full personal apology.

I accept that there are some who will take offence at me addressing this issue under my own volition. I hope the majority will see it as the earnest attempt to resolve this issue as it is, for once and for all.

I have further explanation coming, but for now I have one thing to say. I request that all future references and posts regarding this issue be directed to me. I am sick to death of this issue being used as ammunition to bolster petty feuds on these forum boards. I am taking ownership of this issue, I will work with UWA to make amends at their end, and I will work with the event organizers to realign the event so that such genuine attempts at creativity are welcomed and assessed on merit. I cannot promise specific outcomes. But I promise I will do my best.

Sincerely,

Geoff

[HotRod Todd]

Geoff, you have big balls, and my (amongst many others) respect.

[Z]

I hope the majority will see it as the earnest attempt to resolve this issue as it is, for once and for all.

Geoff,

I also have full admiration for your efforts to resolve this issue.

As with many such situations in life, probably the best way to put an end to it is to do what you are doing here, and give it a full and open public airing. Let everyone involved have their say, back and forth a few times if necessary. Then, with it all "out there", it will quickly become ancient history.

For my part, I was never personally involved in the issue itself. However, my reasons for bringing it up on numerous occasions are NOT to fuel any feud, but more along the lines of "Those who do not learn from History are doomed to repeat it.". Specifically, the treatment of UWA at Oz-12 was, IMO, a serious failure in what is supposed to be an educational competition. Hopefully, by addressing this failure similar such occurances in the future can be avoided.

Here are three ways this competition can be changed to avoid such problems in the future.
~o0o~

1. FEWER POINTS IN DESIGN EVENT - The Design Event is undoubtedly very subjective, and awarding a large percentage of the total 1,000 points to this single Event is inviting trouble. Fewer points = less potential damage.

I think a big part of the "kick in the teeth" to UWA-12 was that in that particular year there were 200 DE points on offer, 50 points more than most other years. Presumably some of those points were available for pure "innovation", but UWA were denied virtually all the points.

In considering by how much the DE points can be reduced, it is a good idea to ask if DE is even NECESSARY to the competition, at all?

For example, consider a competition with Cost worth 200 points, Presentation worth 100 points, and the Dynamic Events making up the rest of the 700 points. Cost is a very important factor in any Engineering project, and as such, is worth quite a bit more than its current 10%. Especially if the scoring is made more objectively realistic than in some past years. Presentation is a very subjective assessment of a Team's "show and tell" skills, but these are a necessary part of real Engineering careers, so arguably are worth 10% of the total points.

But the "design" of the car itself is best judged by the Dynamic Events. So 700 points are awarded for "good design" there, with all these points being objectively, and thus very fairly, judged by a stopwatch.

The DE is often justified with the argument that it is more a test of the students' "design knowledge", than a test of the car itself. But, as the UWA-12 case shows, there are times when the students' knowledge exceeds that of the Design Judges. This makes the DJs incapable of a proper judgement. As this case showed, the result is that the most knowledgeable people at the competition, in that particular area, ended up the losers. It is a general truism that putting "experts" in positions such as those of the DJs, stifles progress.

Another rationale for the DE is that it gives an opportunity for the students to receive educational "feedback" from the DJs. But this feedback only occasionally happens, and is certainly not mandated in any Official Rules. A good example is the almost zero feedback, so far, from the recent Oz-14 DJs (ie. two weeks+ later).

Note that the majority of useful "design feedback" seems to occur wholly outside of the DE itself. Examples are the video reviews of individual Team's cars by Carroll Smith and Claude Rouelle that can be found on the web, and which were given well after the DEs were over. Similarly, the Driver-Swap-Day reviews that have become part of the Oz-Comps in recent years. And, of course, the huge amount of feedback on this Forum.

So, when considering how many points DE should be worth, or even if DE should be part of the comp at all, it is worth remembering that many thousands of years ago some Mediterranean villagers tried living without Kings, Queens, or any sort of Royalty at all. Apparently the resulting "democracy" worked quite well for them. Worth trying again, IMO.
~o0o~

2. LESS SUBJECTIVITY - After following this competition for a decade and a half now, I still have very little idea of what is expected from the students in the Design Event.

The oft-mentioned claim that the DE is a "test of the students' ability to defend their design decisions" sounds, to me, remarkably like the entry test to the Shakespearean Appreciation Society.
"Sooo... you say you lurv Othello, eh? Now, dahlinks, tell us why you think Othello is soooo much better than Macbeth..."

Whichever way I look at DE, I cannot see how the many different Design Judges can make their own personal, and undoubtedly subjective, assessments of the many different cars, and then somehow all come to an objectively fair ranking of the cars.

Specifically, the current guidelines (ie. in the many world-wide variations of the Rules) divide DE into many small sub-categories, with N points awarded to M sub-category. When all the cars happen to be small variations on a standard "cooky-cutter" theme, with each car having components that neatly fit into each sub-category, then there is some chance of this judging process giving a reasonably fair ranking of all the cars.

But imagine a Team that has built a potentially world-beating car, but has done so by focussing almost exclusively on only one of those many sub-categories. Perhaps they have built an Anti-Technology-Special, with most resources devoted to a Mega-DF-Undertray, and hardly any resources to chassis, suspension, engine, electronics, etc. Such a Team would score well in the Aero category, and maybe also in Big-Picture-Thinking. But why would DJs of any of the other categories award good points when the students say "Oh, we've spent NO TIME AT ALL on that stuff..."?

In short, the current guidelines reward a policy of "build a conventional car, then polish all its little bits". But this judging process has no chance of giving a good objective ranking to highly unconventional cars. (This point explained here, from Bob's link in Oz-14 thread.) Any Team contemplating an unconventional car must factor into their overall-point simulations that they might score close to ZERO points in DE (ie. this proven historically by UWA-12).

This then becomes a huge impediment to progress. Especially so, if the number of DE points increases, as some DJs urge.

Anyway, on the Oz-14 Competitions thread I have suggested how DE might be changed to make it more objectively assessable, namely by a calculation of each Team's relative improvement over their previous year's performance. I am sure there are also other ways of providing clear and objective assessments that reflect the "Design" of the car, or of the effort put in by the Team.

But leaving DE up the the subjective opinions of a large number of DJs is always going to have an inherent, subjective, vagueness, especially for unconventional cars.
~o0o~

3. MORE OPENNESS AND TRANSPARENCY - A cliche, but a much needed one, given the opposite direction the competition seems headed in (see below).

At the very least, each Team's pre-comp Design Report, and each Design Judge's post-DE score-sheet, comments, etc., should be made publicly available as soon as possible after the Event. Since each year's car is supposed to be a completely new design I see no reason why the "last year's model" Design Reports should be kept confidential. Making public these Team-DRs, and the DJs' score-sheets and comments, then becomes a very valuable educational resource for all Teams around the world, and should help foster steady progress.

Also necessary, IMO, is that some sort of bio/CV of each of the DJs is made public, listing their areas of expertise, etc. These can then be used by the students when interpreting the scores and feedback. And along with "feedback" going from the DJs to the students, some sort of assessments going in the opposite direction would be helpful.

So as part of an overall post-comp de-brief, each Team should submit a written review of each of the DJs they had, and how educationally helpful they thought those DJs were. In wider (Western) society this process is known as "voting day". It is a useful process for limiting the damage caused by sub-standard Officials.

Summing up this section, it is noteworthy that the recent International Rule Committee's work on the 2015 Rules changes is about as far from O&T as it is possible to get. Sadly, the 2015 Rule changes are atrociously written (ie. great ambiguity+++), and seem to be an almost guaranteed source of future grief. Much of this can be directly attributed to the IRC's anonymity and refusal to engage in open discussions. Namely, a complete lack of O&T.
~o0o~

Apologies if I have gone too far off-topic. Again, the intent of above is to use the "lessons of History" to move the competition more towards Engineering Education, and further away from "Game of Thrones".

I hope that the other players in this UWA-12 issue, especially those on the Official side of things, can post their thoughts here, and help move things to a resolution.

Z

[Claude Rouelle]

Geoff,

Refresh my memory here. Did UWA submit their document on time? If so what was at that time the penalty for late submission? Just want everything to be clear here.....

[Moke]

I am not aiming this at UWA or any other incident in particular, rather at the competition as a whole; teams, judges, officials, scrutineers, volunteers and forum warriors.

Having been on the receiving end of wholesale condemnation from officials in 2005, I know first hand the hurt caused by this sort of treatment. After pretty negative comments from officials, the experience was topped of when jnr team members were called together for a ragging out of snr members by snr international officials over our overambitious designs and how we had done them a disservice by not just rehashing the 04 car. We went back to our motel, cracked some beers and discussed what had been said and our professional options of those officials. 2005 had been a toxic year within the team, very very expensive and we had burned bridges with the uni management. With a change in team management in '06, we decided not to let the experience define us as a team. The exact details of what when down have been forgotten and are unknown by the current team as it doesn't matter compared to the lessons learned.

What has happened can not be changed, nor to be honest should it, but lessons should be learnt by both teams and officials. Transparency, accountability and respect are paramount to preventing repeats of these situations happening again. Teams need to know on what basis they are being assessed before the event and provided with feedback afterwards. But they also need to respect the outcomes if they don't meet the defined criteria. Judges need to be accountable for their assessments of teams designs as teams need to be accountable for their designs and actions.

There is a culture of us vs them, which just leads to more problems than it solves. Teams get shitty and snarky at officials. The officials get shitty at the teams and close down. Having an open dialogue between teams and officials is going to be the most productive way to improve the event.

Oh and full credit to you Geoff.

[Claude Rouelle]

Brent,

Is the professional world you are working in now different?

[Moke]

Brent,

Is the professional world you are working in now different?

In what way Claude? I'm working for a nanofibre R&D and production company. Almost everyday the knowledge of and uses for nanofibre changes. As we are working in the R&D field our goalposts are always being moved and the requirements changed.

[Claude Rouelle]

Brent,

Thank for giving details on your job. Seems to be an interesting and challenging work in a new, promising, rapid developing industry.

I was just wondering if the kind of frustration and happiness you got at the OZ FSAE competition (you speak about toxic year, burned bridges but also transparency, accountability and respect) is something you sometimes experiment nowadays with your colleagues, management, customers or even in your private life with family and friends.

You also wrote: “Judges need to be accountable for their assessments of teams’ designs as teams need to be accountable for their designs and actions” You think judges are not accountable for their assessments? Can you give specific examples of what is missing and what can be improved?

[Moke]

I don't want to get this thread to far off topic as it is starting to go a little down a rabbit hole.

Within our team we had a culture (that I was part of) that if you weren't out at the workshop 24/7 you were a waste of space and that no one would be allowed to stand in the way of our goals. This put us offside with the uni and a number of good sponsors, not to mention that we had a large number of team members quit and never return. From this I learnt how to manage people with different goals, some work because they enjoy it, some work for the money, some for the betterment of the company, some for themselves. Each requires a different approach to motivate. New Zealand, and to some extent the world, is a small place to do business and there is a web of connections that you don't know about. Piss off the wrong person and you are done for. For example at a trade show we had the car at I was talking to a couple of elderly gentlemen for over an hour. After the show I got a call for one of our major sponsors, the gentlemen were his best mates and they were very impressed with us. Now this sponsor had been dicking us around a little, if I had bad mouthed them to these guys the phone call would have been much different. You never know who knows who - so don't burn your bridges.

With regards accountability of judges and teams:
I always found it was hard to get good feedback on the static events, and not just design, from the judges. At best we would get a copy of the marking sheet with very brief notes and normally it would be after the event. If a copy of the marks and comments were handed out at the same time the points were posted it would allow teams to seek clarification to how they were marked. Maybe through out the weekend there are debrief meetings where a team can book a slot with a judge if they wish. Limit the meeting to two or three team members and 15-30min and make it clear that marks will not be changed, it's for the teams to ask questions and listen not to argue.

Teams need to be prepared to present/explain/demonstrate/justify, call it what you will, their design choices, cost reports and business plans, if they are not prepared they need to understand that they will be held accountable and marked accordingly, (again not only talking about 2012 UWA). It is a fundamental of everyone's professional and private lives, that you'll be held accountable for your (in)actions. If I decide to hold off ordering materials and we run out I'm responsible for the lost production, just as I'm responsible if I have excess materials in stock.

[GTS]

In the spirit of covering bases and rolling any constructive feedback into future events (of which we should all hope there are many more);

Given that I had the design process knowledge and, at the time, nearly 12 years experience in Formula SAE, I had sufficient understanding to know that 3 points was not an adequate score. I had spoken to the team both at the event, and prior to that a number of times during the year. I was very impressed with their knowledge and their creativity. Therefore I knew that their final score was not representative of their design knowledge. However I did not take any action to have their score investigated.

BB, I was a visitor at the 2012 event sent at the behest of my then-employer, so this is more a third-hand view (one that admittedly thought the concept quite ingenious if incomplete - I hope the students involved have gone onto bigger things from the experience)...

Was the car complete? (If it wasn't, what would have been C5.11 applies).
Was the submission on-time? (If it wasn't, what would have been C5.9 applies).
Did the car pass rules at time of event (from my vantage point it didn't, and it sounded as though a clarification that should have been sought earlier on, wasn't).
Did the team exhaust its right to protest (A9), and if so, what happened?

Appreciably three points may not appear reflective of a year's work, though some insight into how it came to be would be helpful. Competition points are neither awarded on anecdotal experiences.

For an apology to completely hold water, BB, I'm hoping to next read "I'm 100% sure the car was complete, all submissions were on-time, the car passed all rules way ahead of time, and the team both lodged a formal process and lost unfairly - accordingly, I can 100% justify an injustice occured".

I recognize that the score given was hurtful and even insulting to a team that was willing to take the risk of being innovative. We should be nurturing such inventiveness, for the sake of our future industry.

Before jumping to conclusions that the process stifles creativity or that it was employed in a flawed manner (particularly when many involved value creativity, and some even seek involvement with the event to try to recruit especially creative types - in a competition that's seen much creativity), would we have a deeper insight into what actually happened?

Did the entry fundamentally meet competition requirements? From a third-hand point it wasn't clear that it did.

I believe an injustice occurred under my command

Could you qualify that for us? What do you think the score should have been? Why?

There's space for discretionary rescaling, of course - if this wasn't applied (your post doesn't indicate either way) and you're suggesting it should have been - to what degree? Why?