PROGRAMMABLE TECHNOLOGIES WEB SITE

A scientific study of the problems of digital engineering for space flight systems,
with a view to their practical solution.

MAPLD User Evaluation and Discussion

Dear Conference Attendee,

I would like to thank you for attending this year's conference and taking the time out to fill out the evaluation form.  Your inputs are valued and it helps us put on a productive, enjoyable event.  We have gone through every response.  For the items you were asked to rate, our summer student made histograms and they are included at the bottom of this page.  For your comments, I have typed each one in, categorized it, and responded.  The suggestions were very good and many will be incorporated for next year.  In several places, such as papers on various topics, your assistance is requested.  Together, we can improve and make next year even better.

Again, thanks for taking the time to both attend and fill out the evaluation,

Richard B. Katz
Grunt Engineer
NASA Goddard Space Flight Center

Breaks
Classified_Session
Dinner_Speaker
Entertainment
Evaluation_of_Technical_Content_Topics_For_Next_Year
Facility
Food
Industrial_Exhibits
Invited_Talks
Kudos_and_Overall_Impression
Military_versus_Space
Miscellaneous
Organization
Panel_Session
Peer_Reviewed_Papers
Production_and_Administration
Survey_Results

 

Peer_Reviewed_Papers

We have received a number of requests to upgrade the conference and have formally peer-reviewed papers.  If we do this, this will be the biggest change to the conference that we anticipate.  It is important for many of the authors to have peer-reviewed papers published; additionally, it will raise the level of work at the conference.  We had many good papers this year that are at the appropriate level.   As we also saw during the question and answer periods and other discussions, the input of peers is valuable and makes for a better paper.  That is one of the reasons that final versions of the papers aren't due until two weeks after the conference; the conference attendees are good peer reviewers.

On the negative side of peer-reviewed papers, it places more of a burden on each author.  I don't want to discourage good, interesting talks and this may do exactly that.  Realistically, it is a lot of work and sometimes a hassle to complete a peer-reviewed paper and I have done quite a number of them.  There are other issues involved that complicate matters.  The first is copyright where a journal may require that all authors sign over their rights to the journal.  Another is page charges.   This, from our quick survey, can be quite expensive and we can't afford to bankrupt the conference with it (it's enough for me to get in trouble at work) nor do we wish to raise the conference fee a large amount.  This is an item to be negotiated with the journals and we'll see what makes sense.  An alternative is to pass page charges over to the author's organization.  Most journals that we have looked at don't absolutely require that the page charges be paid but strongly recommend it.

My personal view is that we should have peer-reviewed papers for those that want it.  This doesn't affect my job as my performance appraisal doesn't require publications of this sort - but it does make for better papers.  On the other hand, we don't want to discourage authors.  For example, formatting requirements for the MAPLD '99 Proceedings are given but are optional.  If authors have a template that they are used to, that is fine.  If they use our template (taken from an IEEE society) they will have a very professional looking publication.  With formal peer-reviewed papers, the authors will lose flexibility and have to put more effort in.

As a result, our preliminary plan is to publish future proceedings in two sections: Selected Papers and Papers.  Both oral and poster presentations will be eligible for publishing in the Selected Papers section of the proceedings.  We are also contacting a number of journal editors in the IEEE and AIAA as well as a few other organizations so that we may publish our proceedings as a special edition of their journal.

Putting this together will take a considerable effort.  Dr. Tanya Vladimirova, of the University of Surrey will be the Guest Editor and will become a conference co-chairman.

Panel_Session

Critique of Panel

  1. The panel session was not so good - but the idea is good. Gather many attendees from various fields (designer, system user, quality assurance, etc.).
  2. Panel session good.
  3. Most disappointing panel - Subpar Discussion.  Skip next year or raise quality of panelists some
  4. The panel was well below my expectation, the questions were not address directly. Some panelists were mixing programmability with reconfigurability. Overall I did not get anything out of the panel
  5. Panels were great.

This was the first year we had the panel and obviously there are mixed feelings.   Clearly it needs some improvement.  Attendance at the panel session was fairly high and, in contrast to #3 above, we did have good people on the panel.  It was unfortunate that we did have two panelists drop out because of business conflicts; neither could get out of California to attend the panel session.  One issue was the questions, with a move from a technical discussion (the questions collected) to a more generic discussion.

The panel next year, I believe, should be focused on more technical issues.   That is shown in the suggestions below.  Of course, and this is a theme you'll see here a number of times, it is up to you, the attendee, to make the conference a success.  Please volunteer for the panel, send in topics and specific questions, or suggest good panel members.

Topics for Panel 2000

  1. Pros and Cons of FPGAs for Space Applications.
  2. Design Techniques to reduce power consumption of FPGAs.
  3. Would FPGAs support asynchronous design - if not globally then locally?
  4. 15 years after SPLASH1, why are programming, synthesis, and debug tools still so immature? What should be done?

These all look like good questions for a panel with questions 2 and 3 on the original panel question sheets.  They will be placed on my list for questions for next year.  For the panel, I would like a good mixture of users, industry (devices and software), and academia and the questions, ideally, will cut across the panel.   Again, please send in more questions, topics, and, er, volunteers.

Dinner_Speaker

  1. Dinner speaker good.
  2. MEMs discussion was an interesting talk.

This was our first year with having a dinner speaker and the turnout was good.   Along with the comments here, I received quite a few positive remarks about the dinner speaker.  Therefore, we shall continue this at next year's event.   Additionally, it shall be a technical talk but something different than programmable logic; I think we'll have enough of that over the 2 1/2 days.  I have an idea for a dinner speaker but haven't finalized it or selected a speaker yet.  Please send me your ideas for topics and/or speakers.

Food

  1. The food was too heavy for 12 hours of sitting ... a lighter fare would help. Also, the coffee was terribly weak.
  2. Better coffee.
  3. Need yogurt, bacon, and eggs for breakfast. Note that this attendee rated food a 5.
  4. Excellent meals.
  5. Good food.
  6. Excellent desserts
  7. Bottled water during breaks.

I personally thought the food was better this year than last - and I thought the desserts were, well, dangerous!  I don't drink coffee so I can't comment on that but "I feel your pain."  For next year I shall propose that we make up for our sins this year and offer a number of different coffees for you to select from.  We'll also evaluate the menu for a bit of variety.

7) There was bottled water at every break.

Facility

  1. Excellent facilities
  2. Keep the location.
  3. Keep at APL.
  4. Free phone access.
  5. Free internet access.
  6. Like the conference center, works real good.

Last year we had the conference at NASA GSFC.  While it was designed to fit into Building 8, registrations were high and we had to split between two buildings - luckily we had good weather.  Registration levels increased this year and the NASA GSFC facility would have been overloaded again.  Next year we'll have the conference at APL again, giving us the convenience of the single facility (no walking in the rain) and the ability to hold a classified session.  Both NASA and APL facilities are supplied by our conference sponsors; moving the conference to a hotel would unnecessarily increase the costs.  We have APL reserved for next year's conference, September 26-28, 2000.

This year we provided free phone access for local calls at the end of the industrial exhibits.  Additionally, we supplied a PC with a full suite of office applications and a color printer with materials for making slides and printouts.   Additionally, Internet software was installed and available for all to use.   Programs that were ran include: Word, PowerPoint, Telnet, Netscape, etc., etc.   Next year we shall publicize the facilities better.

Classified_Session

  1. With respect to the classified sessions: you cannot verify a need to know when you don't know what will be presented (besides the title).
  2. Wanted to attend classified session but was never told about the security clearance. <insert unhappy face cartoon>
  3. The location and occurrence of the classified sessions should have been announced - many were confused/walked in late for sessions - particularly the declassified sessions.
  4. Notification by e-mail that clearances were received.

1) The titles of all talks, authors, and an unclassified abstracts were posted on our conference www site.  Additionally, titles, authors, and abstracts were handed to each attendee at check-in.  This volume, titled "Introduction and Abstracts" had this information for each talk in Session F.

2) We will make this clear on the registration form for next year's conference.  The form does ask if you would be going to the classified session and we did put some effort in on cross-checking.  Additionally, we made this clear on the www page for registration, with a hot link to a downloadable form.

3) We announced the schedule during the opening remarks.  Next year this shall be made more explicit by the session chair running the conference in the main hall letting people know what is going on.  For the location of the Session F, it was announced during the opening remarks.  For next year we will augment this with a large sign.  Note that in the programs handed out, we inserted 5-10 minute pauses to allow people to switch between the open and classified parallel sessions.

4) We will ensure positive feedback.  We fixed that for registrations this year, will fix for classified session next year.

Industrial_Exhibits

  1. Exhibits: tear down Wednesday after lunch.
  2. We had an industrial exhibit and had a good flow of traffic with exhibiting being during the breaks.
  3. The exhibits were very relevant and educational.
  4. Ask exhibitions to bring leftover freebies from DAC (smilie cartoon) (including frisbees)
  5. Exhibitor says: Where was NASA attendees, Orbital Sciences?
  6. Add telephone service to the industrial exhibits.

In general, the industrial exhibits were good and we had a good turnout.  This made it a little bit difficult to squeeze them all in.  Location is decided on a first-come, first-served basis.  Based on 2, above, we plan to keep the format of having the exhibits open when oral presentations are not being made.  Based on traffic flow, attendance, and timing, we will shorten the exhibit time and eliminate the one-half day on Thursday.  Those exhibitors who wish to remain open may of course do so.  Closing in the middle of the day Wednesday will hurt, since we saw a number of people who missed the first day as a result of scheduling conflicts. 

I, as a government employee, can not ask the exhibitors to bring anything.   However, everyone can read this page.  Note that government employees can only accept items up to $20 per meeting from a vendor (I consulted our chief counsel on this).   I like the frisbee idea and we hoped to have official MAPLD frisbees for playing on the grass by the lake outside the conference center.  Two problems: 1) It was very rainy; and 2) We can not use conference funds to purchase frisbees.  The conference production is performed by Mary who is a NASA contractor.  This keeps the costs very low but it also puts restrictions on what we can do with the money.

For question 5, that is hard to answer.  I'm sure that moving the conference off of the NASA facility hurt attendance and NASA attendance appears to have been down; overall attendance was up.  Orbital Sciences is another story, we have quite a few people on our mailing list and I handed a supervisor brochures, registration forms, and posters.  I can not explain why we had only one OSC employee in attendance.

For question 6, we will provide telephone service this year.  Most people are now using cell phones and Internet.  Internet service was provided to vendors, at their booth, if they registered early enough.  Phone service can be made available and will next year.  We did not anticipate that vendors would need telephone service to get around their firewall.

Invited_Talks

  1. Brad H./BYU, ESA talk, Rockwell Collins, Xilinx, Katz, et. al. - excellent
  2. Very interesting invited talks in particular.

We kept the same format as last year, starting each session off with 1 40 minute presentation by an invited speaker, and we plan to do this again next year.  This is an opportunity to have some far reaching talks, setting goals and standards, provide a tutorial or history of a topic, or just be interesting and stimulating.  We have a number of invited speakers lined up for next year including Henry Spencer, with whom we're currently discussing talk topics, and Eldon Hall, one of the designers of the Apollo Guidance Computer.  There are still a few spots open or in negotiation and you may send suggestions directly to me.  I hope to get a speaker to talk about spacecraft navigation, gravity assist, and the Voyager and Galileo missions.  With recent events with Mars Climate Orbiter, this may be a bit more interesting than originally planned.

Production_and_Administration

  1. Get corporate sponsors.
  2. Badge - needs to be bigger for name and company.
  3. Fax ok, don't need www/credit card registration [this is from Al!]
  4. Should be publicized even more widely with the aim to gain IEEE support.
  5. August or earlier in September?
  6. The directions never say that the conference was in Kossiakoff Hall.  It only appears on a map without being pointed out.
  7. Helpful to give presentations with laptops.
  8. A very good idea to provide a handout with the presentation slides.
  9. Production: Colors on slides don't always print out in handouts resulting in some "blank" graphs.
  10. All view graphs for papers should be in documents provided - some are not.
  11. Have proceedings CD ready at Conference beginning.

1)  We deliberately do not have corporate sponsors and plan to keep it that way - and do not want to risk turning the conference into a trade show.  This year we did quite a bit better with the talks, keeping them mostly technical.  Part of this was by experience, speakers are understanding that marketeering will be kept in the industrial exhibits (which are provided free of charge).  Part of this was by adding a technical committee, longer abstracts, and reviews of most presentation material before the conference.  Approximately 1/2 of all presentations were subject to re-work, to keep the technical focus.  Three papers were rejected from the conference.  With corporate sponsors, this will become a tougher task.  On the financial side, the cost of the conference is relatively low, as compared with other events, and full meals are provided which is not the norm.  While corporate sponsors may help in advertising the conference, I feel it is not needed and will unnecessarily complicate things.

2) Badges?  We'll try to get bigger names and company names put on them.

3) Registration this year went much smoother than last year with no major problems that I know of.  Last year was complicated by the fact it was our first time and our support contract changed firms on September 1st.  While 3) above doesn't need secure www registration, the majority does, as pointed out by our survey question, and we shall make our best attempt to get this on-line for next year.  This will save us work and provide more accurate e-mail lists.  A service has been identified which is relatively inexpensive; we have to check to make sure that the money handling done by the service and our contractor meets their contract and government rules.

4) Publicity - We have sent out releases to all of the major trade magazines that I know of although admittedly I am weak in military publications.  I also post messages on about a dozen internet newsgroups and pass out posters and brochures to people on our technical committee.  Posters and brochures are also up at several other conferences.   Please send me ideas and places to publicize the conference and I'll make that happen.

5) IEEE support.  This year we had the AIAA co-sponsor the conference for the first time and we did get into Aerospace America, although a bit late to have a major impact.  I also mailed each of the local chapter heads information in the spring.   We are currently contacting the IEEE Aerospace and Electronic Systems Society as that appears to be the most appropriate group for our conference.

6) Directions! One thing that we have discusses is better signs when you get close to APL.  They did put some signs up but we would like them to be a bit more obvious.   Our conference home page and the registration for both identify the Kossiakoff Conference Center; the directions do not.  The directions are a link to existing APL pages.  Next year I shall do our own page, as I did last year, and address this issue.

7) Presentations with laptops.  Last year we supported PC presentations but with the move to APL this year did not.  I could not get everything specified and checked out in time for the Instructions to Presenters release.  There was a lot of new things this year from a production point of view and next year we will support PC presentations.  The timing for the conference is tight as we wish to maximize the number of talks so we will not use laptops and have people fuss with them and attempt to solve configuration problems in real-time.  We found out last year that there are glitches in this system.  Instead, we will supply twin computers (it's aerospace, we have to be redundant) with all of the presentations pre-loaded.  We will also support viewgraphs and 35 mm, as we did this year.

8-11) One of the comments from last year was that we didn't have proceedings ready at conference check-in and that some of the presentations were hard to follow - a lot of detail and for those not sitting in front the writing was too small.  As a good compromise we requested that all presentations be submitted by September 10th.   This gave us time to review the presentations for technical content and printing, so that the attendees could follow along and take notes.  Many modifications were made.  While the instructions to the authors were to ensure that their slides could be printed on black and white paper, a lot did not follow that and I couldn't get all the modifications done, as some presentations were not submitted on-time.  Next year the instructions will be, er, a bit LOUDER with respect to this issue.  A number of presentations came in after I had to submit the majority of them for printing and an Appendix was put together and printed.  Those that were really late, and one individual was printing viewgraphs just about an hour before his presentation, I could obviously not get them printed and distributed.  Since you know who did not submit slides on-time by their absence, you may contact them directly.  :-)  One issue is that some attendees want the proceedings at conference check-in; many of the authors do not.  Additionally, the proceedings will be on CD-ROM which is of limited use at the conference.  From last year's conference we had about 175 people and only 10 requested hardcopy.  The current plan of hardcopy presentations at check-in followed with CD-ROM proceedings is a compromise.

Entertainment

  1. Perhaps, if weather permits, the Wednesday evening meal could be more of a social event like a cookout, etc., with perhaps some recreational activities (volleyball, etc.).
  2. Evening free at least 1 evening.
  3. Only one evening event.
  4. Have cocktail reception 1st night, too. Good for Monday AIAA meeting.
  5. FPGA Contest: Who can build the smallest/fastest (RT, SEU Tolerant)
        - ALU
        - uP
        - etc.
  6. K-Bowl!
  7. Would be glad to have a tour of NASA Goddard Space Flight Center and/or JHU/APL included in the program.

1) I like it!  We shall discuss this with our production people and see if it's feasible and what interest there is.  It looks like we'll do it!  The idea of volleyball, frisbee, etc. will also be discussed.  Was this a suggestion from our "social events co-ordinator?" :-)

2-3) We did have events on three evenings.  On Monday we had a guest lecturer talk at the AIAA chapter meeting; on Tuesday we had an interesting dinner speaker; and on Wednesday we had a panel session.  These events provide an opportunity for all of us to get together and to participate in something interesting.  Additionally, we thought it would be of interest to out of towners.  We are careful, though, to not schedule anything critical to the conference during these times.  I was surprised at the good turnout for all three evenings.  This indicates that we should do this again; we will, however, ensure that the talks are appropriate for an "optional session."

4) We did have a reception Monday evening, from 4-7:30.  If you think other hours would be appropriate, please e-mail me and let me know.

5-6) I did try and set up a contest this year.  However, because of certain circumstances at work, I ran short on time and this had to be dropped.  I talked with some people about doing some contests next year, either based on hardware or on synthesis results.  There are other things, as noted above, that we can do, too.  Are these suggestions from our "contests co-ordinators?"  Thanks for volunteering!

7) We gave tours last year at NASA Goddard Space Flight Center which turned out to be very popular.  I shall look into this for next year, get a bus to travel to NASA from APL (about a 20 minute drive), give the standard tour, and then return to APL for the reception.  Additionally, we can have a tour of the National Security Agency Cryptological Museum (http://www.nsa.gov:8080/museum/index.html).  Please e-mail me with what you would desire.  Tours of APL are difficult since it's a secure facility.

Kudos_and_Overall_Impression

  1. Fills the gap between purely academic and purely marketing forums, been great educational value to both parties.
  2. Great conference. A lot of mingling, exchange, relaxes atmosphere.
  3. Very good job.
  4. Good job overall. This time I was more interested in PLDs.
  5. Good conference.
  6. This is my first time attending the conference. Good cross section of papers.
  7. Excellent conference.
  8. Kudos: Good conference - very applicable and pertinent topics discussed.
  9. Kudos: A very good conference.
  10. A very well organized event getting together academia and the industry on a very hot and promising area of reconfigurable computing.
  11. Very good conference
  12. Very well organized.

This section is of no general use but is good to send to our bosses.  Thanks!

Military_versus_Space

  1. The conference was NASA oriented, not military. This was disappointing because it was not what I expected. On the other hand, it was very interesting to see the NASA applications. In general, the conference was well run.
  2. Maybe increased focus on military applications
  3. More space oriented than conference title would suggest
  4. More military, less space.
  5. Maybe a slight feeling that aerospace was more prominent than military but I was only here for one day.

This is a topic we discussed last year and at the conference this year.  I would definitely like a greater percentage of military papers as one of the reasons for the conference is to get people from the different communities to interact.  To help with this, we do have military people on our Technical Committee.  We can and should do better for next year.  Unfortunately, as a part of NASA, I am in civilian aerospace and my military contacts are a bit limited.  What you can do is help me out; identify people with good material and urge them to submit abstracts; let me know where to publicize the Conference; or write a paper yourself!  We had one drop out this year (visa problem) and we did fill the slot with a very interesting military application.

Evaluation_of_Technical_Content_Topics_For_Next_Year

For the most part these comments stand on their own.  We did take some care in avoiding overlap; in particular with the invited talks.  Not all slides were in on-time and that hurt the process.  We'll try to do better with respect to this next year.

We'll also update our conference brochures and announcements to incorporate these ideas and take them into account in the paper selection process.  Opening a new section is difficult since we have only 2 1/2 days; the attendees quite strongly voted down the move to three days, where we could open up another session.  Additionally, we will need enough papers to fill up these sessions.  We tried to make the overall session titles broad enough to incorporate these things.  Ultimately, though, it comes down to what papers are submitted.  We need your help in identifying people doing work in this area and getting them to submit an abstract.

Organization

  1. Organize the speakers a little better in terms of directions on the podium, slides.
  2. Better layout for poster sessions? Were posters visited by many?
  3. Break it into focus sessions such as applications (mil/space), hardware technology, software programming tools, etc.
  4. Talks were kept to schedule except a couple in D - not every session used the timer. No time over runs like last year.
  5. Parallel sessions - ones such as Military Applications or space applications? [hard to read].
  6. Have a big easel with the schedule posted.

This was our first time at APL and we will work on the organization.  In particular, with the experience we have there now, we can write better instructions for the speakers and slide turners for next year.  There were a few glitches that we can eliminate.

The poster area could have been a bit better layed out.  Our original layout didn't work so well since, when assembled, the posters blocked off too much of the room.   We did have a few congested areas and will give it another look for next year.   One of the limiting issues is the location of outlets used for the lighting on each panel.  Additionally, for next year, we will have the Wednesday morning break dedicated to posters where all presenters will be required to be at their poster.

Parallel sessions are difficult in general and we experimented with one for the first time this year.  While there is merit to parallel military and space application sessions, we did not have enough paper submissions to support that.

We kept pretty much to schedule this year.  Last year a few of the presenters, well, just wouldn't take the hint that their time was up and they abused the privilege.   This year we were determined to be stricter for the presenters time allotment to make things fair and that worked out pretty well.  A few speakers were people who don't normally speak at conferences, and their presentations were a bit slow because of nerves, not microphone hogging.  We do want to be author friendly and speaking for the first time in front of a large audience is not easy so where this was a factor, things were a bit more liberal, with the time made up during the breaks.  We had a good demonstration of the importance we placed on time allotments when I got "the hook" for exceeding my time.  The timer malfunctioned during the conference and the display lights couldn't be set properly, although the timer would run on the control box.

6) Excellent idea, we'll do it.

Breaks

  1. Shorter breaks, too long.
  2. Shorter lunch.
  3. Add breakfast talks!
  4. Long breaks and meals allows good time to interact.

We found that, in both the comments above, and those received verbally, that there are those who desire short breaks and those who like the longer breaks.  With a limited amount of time this is difficult.  Other conferences stop the conference for poster sessions and industrial exhibits.  To save time we mix these activities in with the breaks.  A change for next year is to dedicate the Wednesday morning break for posters.  We also received quite a few comments on how the conference and the facility was a good place for people to interact outside of the talks.  This is important and is something we want to keep.

Miscellaneous

  1. Invite John Cooley (ESNUG)
  2. Conference ended abruptly.
  3. Fax number for evaluation form and on-line form.
  4. Page numbers on papers? Instructions not consistent.
  5. Add more local hotels to the www page.

1) I invited John last year and spoke to him on the phone but no luck.  He's on our mailing list.  If you want him to come, let him know.  I'm sure he would make a good speaker and a fantastic panelist.

2) Next year I'll give some closing remarks.  I didn't want to be a "microphone hog" and bore everyone.  I've been bored at other conferences and want to avoid doing exactly that.

3) Good ideas, we'll do them.

4) The formatting instructions come from IEEE who puts on the page numbers.  I need to update that for next year.

5) Will do.

Survey_Results

 

Do  you plan on attending next year?    AttendNextYear.gif

Which days of the week do you prefer?    DaysOfWeek.gif

How were the conference facilities:        Facilities.gif

How was the food and beverages?    food.gif

How did you hear about the conference?    Howdidyouhear.gif

Industry, government, or academia?    identity.gif

Did the conference meet your expectations?    MeetExpectations.gif

Desire for secure WWW registration with credit card.    SecureWWWRegistration.gif

Should we expand from 2 1/2 to 3 days?    ThreeDays.gif

Home
Last Revised: January 09, 2002
Digital Engineering Institute
Web Grunt: Richard Katz