-
- Browsing
- RSS aggregation for housing news items for Stuart Hodkinson
- Investigating how to generate a list of news items aggregating news feeds on a particular set(s) of topics:
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- MoSeS
- Beowulf processing check:
- Completed 170001-180000.
- Started 180001-190000.
-
- RSS aggregation for housing news items for Stuart Hodkinson
- Stuart asked me to look into how he could receive "all news items on housing".
- Clearly "all news items on housing" is more than anyone wants, but I know what Stuaert wants:
- He wants a list of news feeds on a particular set(s) of topics, but he doesn't want to be told about all feeds, he just wants to be allerted to items that contain information/news about housing.
- Houseprice data
- Geomorphometry
- Results generated for Alice Milner's study region in Greece.
- Uncompressed the files are 28.1 GB (30,252,296,314 bytes).
- Compressed into a zip with 7Zip software using default options it is 6.67 GB (7,169,732,262 bytes).
- Some example output:
- Steep Slopes - More Spur Like
- Steep Slopes - More River Profile Like
- Saddles
- Steep Planar Slopes
- River Basins Processes and Management Research Cluster
- Lee Brown asks what the geomorphers think of the following article:
- Seminar
- Subsoil effects of elevated atmospheric CO2
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Added comment on SAK-12262.
- Added comment on SAK-12262.
- MoSeS
- Edited the NCeSS portal MoSeS Worksite Wiki Home Page:
- Wiki syntax:
-
h2 {worksiteinfo:name} Worksite Wiki
This page is the [Home] wiki page for the {worksiteinfo:name} worksite which was set up on 2007-11-21. A URL to its content is as follows:
http://weaver.dl.ac.uk/portal/site/moses/page/6bd1f652-e028-4e04-8092-f4605dcfeaad/home.htm
h3 People
* Alphabetical list of People by Surname, Firstname
** [Junaid Arshad]
** [Mark Birkin]
** [René Jordan]
** [Dianna Smith]
** [Paul Townend]
** [Andy Turner]
** [Belinda Wu]
** [Chengchao Zuo]
h3 Documentation
* [Meetings]
h3 Recently changed pages
* {recent-changes}
h3 Worksiteinfo
* worksiteinfo:wikispace {worksiteinfo:wikispace}
* worksiteinfo:name {worksiteinfo:name}
- Beowulf processing check:
- Received and written result with 46 HSARDataRecords and 0 ISARDataRecords for 60QC000402 with fitness 5.74291518883563404790
- Sending input 179486 with ZoneCode 60QC000434 to 22
- Browsing
- CSAP Seminar
- Multi-level modelling
- Myles Gould
- Quantitative research classified
- Extensive v Intensive
- General v Specific
- Nicky Best et al (1996) Modelling Realistic Complexity
- Goldstein and Silver (1989) and Mollering and Tobler (1972) argued that societies have a built in hierarchical structure.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Seems like GENESIS is going to get the go ahead :-)
- MoSeS
- Housing Meeting
- Edited the Old (First) NCeSS Wiki MoSeS page
- Wiki Code
-
Modelling and Simulation for e-Social Science
- This Wiki is being decomissioned!
* On 2007-11-27 Andy Turner could still edit this page.
* The information is to be made availabel on a new wiki in due course.
- Links to main MoSeS information sources
* [http://www.ncess.ac.uk/research/geographic/moses/ NCeSS MoSeS Web Page]
* [http://intranet.ncess.ac.uk/bscw/bscw.cgi/3257 NCeSS Intranet MoSeS Content]
* [http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/ Andy Turner's MoSeS Web Page @ School of Geography, University of Leeds]
- Events
* [[MoSeS GIS and Grid Computing ASW]]
* [http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/events/LaunchEvent/ Andy Turner's MoSeS Launch Event Page]
- Collaboration
* As a node of NCeSS, MoSeS aims to collaborate in an open way with other NCeSS initiatives and should look to collaborate generally, and especially in terms of demographic model building and its applications/use.
** If you are interested in collaborating. Please contact us.
* [http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/collaboration/ Andy Turner's MoSeS Collaboration Web Page @ School of Geography, University of Leeds]
* MoSeS overlaps with the [http://www.jisc.ac.uk/whatwedo/programmes/eresearch_grid_ogc_collision.aspx OGC Grid Collision] and the [[NCeSS e-Infrastructure Project]].
- Beowulf processing check:
- Received and written result with 106 HSARDataRecords and 49 ISARDataRecords for 00PTPG0036 with fitness 4.78789284470624602174
- Sending input 175288 with ZoneCode 00PTPH0025 to 27
- Browsing
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Created an issue on the Sakai JIRA.
- Workflow, portals, security e-IP Meeting
- Meeting with Junaid about Work
- Security is a bit of a hot topic in the news!
- Junaid offered to report on a monthly basis about the work he is doing for NeSS:
- This seems sensible and I welcome such a thing.
- Some summary/details of these reports (including dissemination) to be fed into MoSeS Node quartely reports via Mark Birkin.
- We are to make good use of the NCeSS portal and put up information on the relevant wiki's.
- It is important for Junaid to have Web Presence in the form of a Distributed Systems Personal Home Page at the following URL:
- I am not sure if Junaid will also get a School of Geography Home Page. If he does, it should probably be very basic and point to the above.
- SoSoRNet...
- Publications...
- Junaid's MSc thesis has novel ideas on how to define security requirements for e-Social Science applications.
- Junaid has been looking into OGSA-DAI and is looking into SRB
- Junaid Actions:
- Get a home web page put up at the following URL:
- Add/develop wiki pages about him on the NCeSS Portal NCeSS e-IP and MoSeS Worksites.
- Review OSS Watch Wiki: Integrating Sakai and Shibboleth
- Progress collaborations on Security e-Infrastructure in the e-Research community.
- Continue looking into SRB and OGSA-DAI.
- Andy Actions:
- Complete reading of Junaid's MSc Thesis.
- MoSeS
- Beowulf processing check:
- Completed 160001-170000
- Started 170001-180000
- Browsing
- NGS
- Chat with Steven Pickering:
- I told Steven what Shiv and I had been upto and what we were planning for MPJ Express jobs across all NGS core sites.
- Steven was of the opinion that we might be trying to do things the old way and that we might not have thought of everything:
- What if another users job jumps on a resource between the node list generation and MPJ Express environemnt initialisation?
- What if the pbs job scheduler juggles the jobs?
- I should probably get back to Shiv and talk to Jason Lander.
- One of my jobs did die mysteriously, so I wonder if that is symptomatic of some of what Steven raised.
- Communications from Shiv:
- I should probably submit another NGS Renewal Application for MoSeS work.
- Further NGS Renewal Application for MoSeS work:
- Renewal Case:
- CPU hours 50000
- I have used 8400 hours of CPU well over my initial 1000 hours allocation. I have an application in process for a further 10000 hours CPU, but have been advised to submit a further application. This I hereby do. This allocation is for MoSeS work (http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/). In this we are creating a demographic model of the UK. To begin with I am performing a Population Initialisation, which is creating an individual and household level population dataset for the UK for 2001. This is a datset with approximately 60 million records. The dataset is constructed in parts for Census Aggregate Statistics Areas. The lowest level aggregation of these are Output Areas of which there are 223060 in the UK. It takes approximately 5 minutes to calculate the population for an Output Area on a single processor. I hope to use NGS resources to generate a dataset in much less time than it would take on other resources that are available. At the same time I hope to collaborate with NGS and middleware developers to support our use case, which might ultimately involve repeating this sort of process multiple times. At the present time I have been processing on ngs.leeds.ac.uk, but the hope is to submit jobs via the NGS or NCeSS portal and not have to worry which of the core sites the job is running on. All of the details of our research are to be made available on the Web. Progress on this work will probably be presented at the next UK e-Science All Hands Meeting and International e-Social Science conference.
- Renewal Case Summary
- I want more resource to run the MoSeS demographic model population initialisation. This is e-Social Science work to be presented at the next UK e-Science All Hands Meeting and and International e-Social Science conference. Publication of this work will also be on the web and linked via the following URL: http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/
- Unfortunately, when I submitted the renewal form my browser was pointed to the following URL:
- http://ngsdb1-vip.rl.ac.uk:7778/pls/htmldb/wwv_flow.accept
- The following is an image of what my browser displayed:
- I have contacted Shiv to see if he can help with this.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Browsing
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- MoSeS
- In collaboration With Xiaobo Yang set up a MoSeS Worksite on the NCeSS portal :-)
- This is currently available via the following URL:
- This should probably be available via the following URL:
- Beowulf processing check:
- Received and written result with 114 HSARDataRecords and 0 ISARDataRecords for 00NXPA0029 with fitness 9.78688619925708596351
- Sending input 169774 with ZoneCode 00NXPB0013 to 21
- NGS processing check:
- I have exceeded my account limits on the NGS for CPU. My current usage is usage = 6492.894 and your resource allocation is allocation = 1000.
- My account will be locked until I apply (and I assume granted) further resources.
- I made an application on
- Browsing
- MASS
- MASS Meeting
- NCeSS Portal Wiki Edits
- Home
-
h2 Welcome to the Wiki Tool
This page is your [Home] page for this worksite. You can change it by clicking on the edit link above.
h3 People
* [Andy Evans]
* [Kirk Harland]
* [Nick Malleson]
* [Dan Olner]
* [Kim Proctor]
* [Andy Turner]
* [Belinda Wu]
* [Dimitris Kavroudakis]
h3 Meetings
* [MASS Meeting 2007-11-20|MASS_Meeting_2007-11-20]
h3 Links
* Internal Wiki Links
** {link:Multi-Agent Systems and Simulation Research Group|worksite:/Web Links/http:__www.geog.leeds.ac.uk_groups_mass_}
h3 Recently changed pages
{recent-changes}
- MASS meeting 2007-11-20
- h2 MASS meeting 2007-11-20
h3 People
* [Nick Malleson]
* [Dan Olner]
* [Kim Proctor]
* [Andy Turner]
h3 [Dan Olner]'s PhD
* Is it possible to re-examining the 20th century 'economic calculation debate' using 21st century computational methods?
** Paper document - an electronic version of this could be put in resources and linked here?
** Now we have computational models and data that can be used to construct models.
** Does Dan need a migration model?
*** Migration is hard.
** How much of the system needs to be modelled before simulations from the model yield useful/definitive result.
h3 [Nick Malleson]'s PhD
* Can we develop an agent based model to predict burglary rates in a part of Leeds
h3 What [Kim Proctor] is up to
* Looking at Cancer
** Obesity
** Food/Diet
** Environment
*** Pollutants
**** Pesticides
**** Andy Turner mentioned the eSheet proposal developed by Haibo Chen to look at exposure to traffic pollution
** Not much spatial work has been done
*** Danny Dorling has done some on lung and stomach cancer and deprivation
*** Stan Openshaw did work in the early days of GIS developing Geographical Analysis and Explanation Machines
** Spatial Analysis
*** Scan Statistics
*** Clustering/hotspot detection
*** {link:GAM: Cluster hunting software|http://www.ccg.leeds.ac.uk/software/gam/}
** Data
*** National Cancer Register
**** Individual level Cancer data for the UK
**** Has names and addresses
*** Andy Turner still thinks that census data and such registers should be pseudo ananymised and linkable for GAM type analysis
**** What does it take to get this done?
h3 Sakai, NCeSS portal and other Virtual Laboritories
* No demo
** Nick and Kim want to have accounts
** Andy to email others again to ask if they want accounts for the NCeSS portal
- Andy Turner
-
h2 Andy Turner
* {image:sakai:/~a.g.d.turner@leeds.ac.uk/images/a.turner.png|a.turner.png}
* Internal Link
** This sometimes works and sometimes is broken!
** {link:Andy Turner's Home Web Page @ School of Geography, University of Leeds|sakai:/~a.g.d.turner@leeds.ac.uk/Web Links/http:__www.geog.leeds.ac.uk_people_a.turner_}
* External Link
** {link:Andy Turner's Home Web Page @ School of Geography, University of Leeds|http://www.geog.leeds.ac.uk/people/a.turner/}
h3 Brief Description
* Works on the MoSeS Node and the [NCeSS e-Infrastructure for the Social Sciences Project|Home].
h3 References
* Internal Links:
** These sometimes work and sometimes are broken!
** {link:Multi-Agent Systems and Simulation Research Group|worksite:/Web Links/http:__www.geog.leeds.ac.uk_groups_mass_}
** {link:NCeSS e-Infrastructure for the Social Sciences Project Home Web Page|worksite:/Web Links/http:__www.ncess.ac.uk_services_research_}
* External Links:
** {link:Multi-Agent Systems and Simulation Research Group|http://www.geog.leeds.ac.uk/groups/mass/}
** {link:NCeSS Services Research|http://www.ncess.ac.uk/services/research/} - (NCeSS e-Infrastructure for the Social Sciences Project Home Web Page)
** {link:Andy Turner's NCeSS e-Infrastructure for the Social Sciences Web Page @ School of Geography, University of Leeds|http://www.geog.leeds.ac.uk/people/a.turner/projects/eInfrastructure/}
** {link:Andy Turner's Home Web Page @ School of Geography, University of Leeds|http://www.geog.leeds.ac.uk/people/a.turner/}
** Links to other Andy Turner NCeSS Wiki Pages
*** {link:My Workspace Wiki Home Page|http://weaver.dl.ac.uk/portal/site/%7Ea.g.d.turner%40leeds.ac.uk/page/703d10f9-dc56-4305-009a-9b3f8479dd42}
*** {link:ESRC e-IP Andy Turner Page|http://weaver.dl.ac.uk/portal/site/18e05b54-6c2e-4963-80a1-b56ef16ec87b}
*** {link:Information Andy Turner Page|http://weaver.dl.ac.uk/portal/site/88865e79-474e-4b91-8071-049e0a6282aa}
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Completed e-Research Survey on request of Eric Meyer and William Dutton
- They asked while I took the survey, to make a note of any questions that 1) you think we should cut, 2) you think should be changed, or 3) have some sort of technical or other problem.
- Also, they asked for comments if there are important questions you feel we should add, or with any general comments about the survey to help them prepare to distribute it more widely.
- I have tried to provide these inline below and also at the end of this item. These might be easier to find in the source of this page.
- A researcher is a researcher, why bring PhD's into it?
- I don't see how the question about single versus co-authored work can be relevant!
- Perhaps RSS news feeds should be added?
- In the recent past I usually used the telephone for audio-conferencing:
- Perhaps that should be a selectable option.
- In this it was best to have a speaker phone set up and to take the call in my office whilst working on my PC.
- I have recently started making less use of Access Grid Facilities and more use of my PC for most virtual meetings.
- With this I am still using Access Grid software.
- For the video-conferencing device question. I selected that I use a tool and the list of options are software. The word device is confusing. I think the device I use is a networked web cam, but that is not the information you want...
- Database software is often tightly coupled with other software.
- It can be unknown how many researchers and developers you are collaborating with using open source software.
- My answer is based on those who are getting a share of the project funding.
- I would add wiki to the tool list in the usefulness question. Indeed these questions could have the same list of options.
- I'm not sure about use of an Institutional Repository (IR)
- I'm not sure really what an IR is.
- I think some of the work I have done as part of European Commission funded work has found its way onto their repository.
- Is the School of Geography website at the University of Leeds an IR, or part of one?
- References
- General Feedback:
- I made an effort to capture my answers to the survey as seen above. It would be good to know that I would receive this in some form after submitting the survey prior to completing it. That would make it easier for me to blog :-)
- MoSeS
- Beowulf processing check:
- Received and written result with 149 HSARDataRecords and 0 ISARDataRecords for 00NQPX0005 with fitness 10.22027634670329548839
- Sending input 168591 with ZoneCode 00NQQB0007 to 18
- NGS processing check:
- java.lang.NegativeArraySizeException
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run_OA(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:225)
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:114)
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.main(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:71)
- Browsing
- Some thoughts on Distance Weighting and Raster data generalisation:
- Within a distance (in a region) the mean and variance of the distance of the cells (raster measurements) can be calculated:
- This can be used to construct a normal distribution weighting scheme:
- The weight at any point can be calculated using:
- Considered registering for Workshop on Combining macro and micro data from the census.
- I emailed to try and find out who is involved in running this workshop so I could ask if the work I have been doing on developing individual, household and communal establishment population data for 2001 is of interest.
- This work has involved selecting sets of Indiviudal and Household SAR records to populate Census Aggregate Statistics Areas.
- Some of this work is being done on a UK scale.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- MoSeS
- Loading CAS003DataRecords for Northern Ireland:
- Northern Ireland table:
- England Table:
- Is this really necessary?
- Well, the following variables are used to constrain the current population initialisation for all other areas:
- HRP Age 0 to 19
- HRP Age 20 to 29
- HRP Age 30 to 39
- HRP Age 40 to 49
- HRP Age 50 to 59
- HRP Age 60 to 69
- HRP Age 70 to 79
- HRP Age 80+
- Can these variables be got from other tables?
- CAS005 Age of HRP by Sex and Living Arrangements
- This has similar age breakdown problem to CAS003.
- CAS067 Age of HRP and Dependent Children by Approximated Social Grade
- No use, just a dichotomy around age 35!
- CAS327 Age of Household Reference Person (HRP) and Tenure by Economic Activity of HRP
- This has similar age breakdown problem to CAS003.
- It seems not unless there is a commissioned table:
- A common aggregation for the UK is having Age 30 to 59 as a single constraint group.
- At least, that is what is probably wanted for a UK dataset based on identical constraints for each area.
- Unless we allow consistent to mean use of the same Census data constraining tables.
- The only other option (I can think of at present) is to commission a table.
- Had a quick meeting with Mark about this:
- I will look into reconfiguring runs for Northern Ireland so that it uses the 15 year age groups for Ages 30 to 59:
- There is a slight issue in doing this in that the Age of People in the Household SAR is in two year aggregations and these don't fit with a split on age 45:
- There are a few ways to cope with this:
- One is to ignore it.
- Another is to treat HRP's with age 45 as both 44 and 45 year olds:
- This can be done by:
- duplicating records
- pre-processing age assignment (disaggregation)
- some persistent (control constraint consistent) random age assignment
- The easiest is to ignore this although in discussion with Mark the plan was to duplicate records.
- Beowulf processing check:
- Received and written result with 117 HSARDataRecords and 0 ISARDataRecords for 00NJPU0002 with fitness 13.85105151982409507305
- Sending input 167125 with ZoneCode 00NJPY0001 to 10
- So, it hasn't fallen over, but when I checked, it is throwing lots of errors as a result of CASKS006 not being loaded for Wales.
- This is creating a large log file, but the results seem to be being generated fine :-)
- NGS processing check:
- java.io.EOFException
- at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
- at java.io.RandomAccessFile.readLong(RandomAccessFile.java:758)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataRecord.(CAS003DataRecord.java:1030)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataHandler.getCAS003DataRecord(CAS003DataHandler.java:247)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataHandler.getDataRecord(CAS003DataHandler.java:233)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataRecord.init(CASDataRecord.java:176)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataRecord.(CASDataRecord.java:147)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataHandler.getDataRecord(CASDataHandler.java:362)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run_OA(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:261)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:114)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.main(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:71)
- Problem in getCAS003Record(RecordID(218038))
- java.io.EOFException
- at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
- at java.io.RandomAccessFile.readLong(RandomAccessFile.java:758)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataRecord.(CAS003DataRecord.java:1030)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataHandler.getCAS003DataRecord(CAS003DataHandler.java:247)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CAS003DataHandler.getDataRecord(CAS003DataHandler.java:233)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataRecord.init(CASDataRecord.java:176)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataRecord.(CASDataRecord.java:147)
- at uk.ac.leeds.ccg.andyt.projects.moses.io.CASDataHandler.getDataRecord(CASDataHandler.java:362)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run_OA(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:261)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.run(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:114)
- at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_NGS.main(IndividualCensus_SWR_HSARHP_ISARCEP_NGS.java:71)
- mpiexec: Warning: tasks 0-4,17-92,97-100,109-127 died with signal 9 (Killed).
- mpiexec: Warning: tasks 5-16,93-96,101-108 died with signal 15 (Terminated).
- This is because there is an Incompatible Table Layout for Northern Ireland for this table. I now need to know what to do about this. I should probably have worked this out earlier!
- Job 38681 in the Q to process with 128 nodes RecordIDs 190001 to 200000.
- Browsing
- :-) I have a work laptop now :-)
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- [mpj-user] Running MPJExpress on the NGS
- Submitted the following NGS Renewal Application for MoSeS work:
- Renewal Case:
- CPU hours 10000
- I have just started using my previous allocation for MoSeS work (http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/). In this we are creating a demographic model of the UK. To begin with I am performing a Population Initialisation, which is creating an individual and household level population dataset for the UK for 2001. This is a datset with approximately 60 million records. The dataset is constructed in parts for Census Aggregate Statistics Areas. The lowest level aggregation of these are Output Areas of which there are 223060 in the UK. It takes approximately 5 minutes to calculate the population for an Output Area on a single processor. I hope to use NGS resources to generate a dataset in much less time than it would take on other resources that are available. At the same time I hope to collaborate with NGS and middleware developers to support our use case, which might ultimately involve repeating this sort of process multiple times. All of the details of our research are to be made available on the Web. Progress on this work will probably be presented at the next UK e-Science All Hands Meeting and International e-Social Science conference.
- Renewal Case Summary
- I want more resource to run the MoSeS demographic model population initialisation. This is e-Social Science work to be presented at the next UK e-Science All Hands Meeting and and International e-Social Science conference. Publication of this work will also be on the web and linked via the following URL: http://www.geog.leeds.ac.uk/people/a.turner/projects/MoSeS/
- MoSeS
- Beowulf processing check:
- Received and written result with 141 HSARDataRecords and 0 ISARDataRecords for 46UCHD0003 with fitness 72.86766015756528671591
- Sending input 162727 with ZoneCode 46UCHF0005 to 20
- I'm expecting this to fall over trying to send 165665 - the first OA in Wales as KS006a is not loaded in CASKS006.
- I have fixed the code on my PC, but this is yet to be shifted to the Beowulf.
- CASWEB Table Layouts for KS006
- England
- Wales
- Scotland
- Northern Ireland
- NGS processing check:
- Job 38019 in the Q to process with 128 nodes RecordIDs 200001 to 223059
- A estimate is that this will take about 15 hours once started.
- 23058 OAs, 5 minutes each, 128 nodes ~ ( 23058 * 5 / 128 * 60 )
- Browsing
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- NCeSS Portal
- Aranged with Rob Allan to add MASS as a Worksite or Workspace.
- Wiki links:
- NeSS Project meeting via Access Grid
- Andy Turner's NCeSS e-Infrastructure Project Meeting 2007-11-15 Web Page
- Agenda
- Wei Jie presentation on authentication/authorisation service for the e-Infrastructure
- Opportunity for everyone to offer feedback and help with use cases.
- This should be of particular interest to those involved in work for the e-Infrastructure that have security requirements.
- People
- Mike Daw (Chair)
- Wei Jie (Presenter)
- Rob Allan
- Andy Turner
- Junaid Arshad
- ...
- Documentation
- Links to Portal Resources
- ...
- Notes
- I had problems connecting, but the problems were more at the server end it seems. I tried unicast and multicast and probably should have tried a bridge, but didn't. For detials see the following:
- Wei presented and Junaid and I joined the meeting from my office and we tried to catch up.
- In the feedback session:
- I pointed to Junaid's MSc thesis which I had uploaded on Tuesday.
- Wei said he had already read it.
- Junaid made some points along the following lines:
- One of the problems we are facing in MoSeS is the confidential nature of the data and that Security of the data itself did not seem to be being addressed.
- In Junaid's MSc Thesis he advocated integrating PERMIS and Shibboleth.
- GridShib and ShibGrid are projects integrating Shibboleth and Globus.
- OGSA-DAI is part of Globus which is built on GSI built on PKI with inherent limitations.
- Wei is to meet with MoSeS (Junaid, Paul Townend, Andy) to talk about Security:
- Junaid strongly encouraged collaboration.
- This could be part of the same meeting as when Rob Allan and Xiaobo Yang come to Leeds.
- It has been suggested that a date in early December would be good, but this is yet to be arranged.
- I lost connection towards the end of the meeting.
- After the meeting Junaid and I had a chat:
- It would have been good to catch up before this meeting so that I could brief Junaid about the meeting on Tuesday.
- Junaid does not yet have access to the NCeSS e-IP Worksite on the portal.
- Andy is to sort this out.
- Junaid still does not have a web home page:
- Paul is to sort this out.
- Junaid has talked to Paul and plans to investigate whether SRB is an alternative to OGSA-DAI.
- We know SRB is a virtual file system, but is it more and what does it have in terms of security?
- Slide 22 is vague and there is a contrary typo.
- In terms of MoSeS work, Junaid is of the opinion that moving from Gridsphere to Sakai is not a security problem.
- That is principally right because both are based on tomcat, however:
- Sakai requires a dedicated tomcat instance and Shibboleth and PERMIS tie tomcat to apache.
- The potential issues being multiple apache webservers on the same resource.
- This might be a Red Herring, but Junaid is going to look into it.
- References
- Why I was experiencing problems and arrived late:
- MoSeS
- Beowulf processing check:
- Received and written result with 113 HSARDataRecords and 26 ISARDataRecords for 45UFGP0016 with fitness 1.90466020765621845464
- Sending input 161454 with ZoneCode 45UFGR0004 to 1
- Browsing
- [OSGeo-Discuss] idea for an OSGeo project -- a new,open data format:
- MASS
- Setting up a wiki and using a virtual research environment:
- There have been calls for the setting up of a wiki within the MASS group.
- I asked during lunch at the meeting in Lancaster yesterday if the NCeSS portal could support MASS and Rob Allan was willing to help. I sent the following email to Mass and Rob today:
- I think we would benefit from using the NCeSS portal (http://www.ncess.ac.uk/portal) and having MASS set up as a Sakai Workspace. I asked Rob if this was something for MASS types yesterday and he is happy to help us. What we need to do is provide him with a list of email addresses and he will sort out accounts. What is available is much more than a Wiki, it is a full on virtual research environment and I'd really like to encourage us to use this option and use the available Tools. As users, we can feedback on the features to encourage them to evolve into what we want. As I understand it, this would not only provide MASS with a Wiki etc, but it would be indirectly helping NCeSS and related organisations.
- UoL
- Re-registered for the University of Leeds VPN via the following URL:
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- Go-Geo metadata creation
- Emailed research postgraduates in the School of Geography:
-
- Based on Ant's well crafted text:
- Requested item in School of Geography Newsletter based on the above text.
- Reply from Ulrika...
- MoSeS
- Beowulf processing check:
- 150001 to 160000 seems to have completed.
- Starting 160001 to 170000.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences (NeSS)
- Browsing
- Grids
- Consider idea over the weekend to handle StackOverflowErrors
- In init_MemoryReserve(AbstractGrid2DSquareCell,ChunkID,boolean) I had changed:
- HashMap _HashMap = _SwapToFileGrid2DSquareCellChunkExcept(
_Grid2DSquareCell,
_ChunkID );
- to:
- HashMap _HashMap = _SwapToFileGrid2DSquareCellChunksExcept(
_Grid2DSquareCell,
_ChunkID );
- because of a runtime StackOverFlowError encountered when trying to handle OutOfMemoryError.
- I wonder if this is still the case...
- If it is can the StackOverFlowError be handled sensibly with a catch?
- Preparing for meeting in Lancaster tomorrow:
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- Meeting with Shiv to get MoSeS code running on NGS resources:
- Shiv has developed a C program that generates a list of nodes (machines) that are used to stage an MPJExpress job on an NGS resource:
- The program is called makeconf and it passes it is used to inject a list of machines for MPJExpress environment initialisation.
- There are two configuration files used in running a process:
- launch
- This sets up the platform and the last line can be modified to run different compiled java class or Jar files.
- pbs_job
- The second line of this controls the size (number of parallel jobs) - in the following example there will be 128:
- Steps to running the job:
- Get UK e-Science Certificate.
- Download and run the GSI-SSHTerm Application
- File : New Connection: ngs.leeds.ac.uk
- Host: myproxy.grid-support.ac.uk
- Account Name: geoagdt
- Point to certificate:
- Tools: SFTP: upload data files>
- Copy the mpj staging files (from temporary location where Shiv made them available).
- Configure number of processors and java command in launch and pbs_job files (as described above).
- sub pbs_job
- keep a note of the job number
- Watch job Q and download output once job completed.
- The nodes list is written to a file mpj.conf.
- A log file is written starting with pbs_job.0 and ending with the job number.
- A list of useful commands:
- watch qstat
- This watches the Q status of submitted jobs.
- qdel "job number"
- MoSeS
- Provided brief report to Mark on MoSeS activities over the last 3 months (Aug-Sep-Oct) including a note of any meetings attended, presentations given, etc:
- Andy Turner's Report on MoSeS Activities 2007-08-01 to 2007-10-31
- I was away in August on Annual Leave and in September on Paternity Leave.
- During this period progress has been made with the UK Population Initialisation although we have been beset by problems with the stability of the Beowulf cluster machine we have been using.
- This period also saw the publication of work done on the OGC GeoLinking IE which I had a hand in.
- Paul demo'd this at the AHM and in OGF.
- Progress has also been made with the NCeSS e-Infrastructure for the Social Sciences Project.
- I have been project managing this and preparing for this work which can be regarded as MoSeS work and progress.
- 2007-10-31
- Meeting with Shiv Kaushal and Paul Townend to discuss using NGS resources for MoSeS modelling work
- The population initialisation is approximately 200000 chunks of processing that can be done concurrently (in parallel).
- Plan to get MoSeS Population Initialisation code running on NGS resources in the next few weeks.
- 2007-10-30
- 2007-10-22 - Ongoing
- Developing program to patch all Beowulf results together whilst checking for missing results.
- 2007-10-03
- 2007-10-03
- 2007-08-14
- Meeting with Paul and Junaid about Junaid's MSc project work.
- 2007-08-10
- 2007-08-07
- Published outputs for IPS GA Comparison:
- 2007-08-01 to 2007-08-10
- Developing code for producing regression reports of IPS and GA comparison runs.
- Provided brief report to Mark on what Junaid's been doing that is relevant to MoSeS:
- Junaid has only just started in post looking at security.
- This follows on from his MSc Thesis work done over the summer.
- Thesis attached to communication.
- Beowulf processing check:
- Received and written result with 135 HSARDataRecords and 3 ISARDataRecords for 43UKGL0003 with fitness 8.19944646411474425992
- Sending input 157292 with ZoneCode 43UKGM0006 to 2
- Browsing
- Meeting with Ant about Go-Geo metadata creation
- Tax concerns of incentive scheme
- Training courses
- Getting people interested the key...
- Preparing some Blurb (publicity material):
- Paraphrase Go-Geo's original email.
- Explain why we jumped at the opportunity.
- Outline incentive scheme.
- Ask for feedback if interest.
- Also ask for feedback if interest in a training course.
- Send Blurb to EDINA on Tuesday to check all is OK
- Send Blurb to Geography PhD students first to gauge interest in a training course
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- MoSeS
- Prepare brief report to Mark on MoSeS activities over the last 3 months (Aug-Sep-Oct) including a note of any meetings attended, presentations given, etc:
- Prepare brief report to Mark on what Junaid's been doing that is relevant to MoSeS:
- Beowulf processing check:
- Received and written result with 147 HSARDataRecords and 0 ISARDataRecords for 42UDFU0005 with fitness 19.85089216501287528078
- Sending input 152759 with ZoneCode 42UDFU0025 to 3
- NCeSS Portal Usability Workshop, Manchester, UK
- Browsing
- Debugging Grids:
- Still having problems processing José's data - Geomorphometrics Use Case:
- OutOfMemoryError handling still not working correctly since code refactoring to expose the handling at a higher level.
- Debugging Grids:
- Still having problems processing José's data - Geomorphometrics Use Case:
- OutOfMemoryError handling still not working correctly since code refactoring to expose the handling at a higher level.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- Evaluating SAW-GEO
- Supervision meeting with Junaid
- Primarily for organising Junaid's attendance at the Portal Usability Workshop in Manchester 2007-11-08.
- Junaid wanting to get root access on one of the cslinr20 nodes for experimental work.
- Briefed Junaid on what is going on with regard computation on NGS resources as being organised by Shiv and Paul.
- Junaid has a UK e-Science certificate.
- I am to pass on link Shiv sent re: getting more time/space/etc on the NGS:
- Browsing
- Debugging Grids:
- Still having problems processing José's data - Geomorphometrics Use Case:
- OutOfMemoryError handling still not working correctly since code refactoring to expose the handling at a higher level.
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- Evaluating SAW-GEO
- MoSeS
- Beowulf processing check:
- Processing seems to have halted prematurely:
- 04:45:32.697 EVENT Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=15000]
- 04:45:32.699 EVENT Stopped SocketListener on 0.0.0.0:15000
- 04:45:32.699 EVENT Stopped HttpContext[/]
- 04:45:32.699 EVENT Stopped org.mortbay.http.HttpServer@1434234
- 04:45:32.699 EVENT Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=15001]
- 04:45:32.700 EVENT Stopped SocketListener on 0.0.0.0:15001
- 04:45:32.700 EVENT Stopped HttpContext[/]
- 04:45:32.700 EVENT Stopped org.mortbay.http.HttpServer@186768e
- Received and written result with 98 HSARDataRecords and 0 ISARDataRecords for 40UBHS0003 with fitness 7.83268819581763672618
- Sending input 148107 with ZoneCode 40UBHW0007 to 11
- node102 seems to have died.
- node102 was rebooted and it's memory swapped to 2x1GB and 2x128MB modules:
- There was a test for having 2x1GB and 2x512MB modules in node101-103 but it would appear that the old 512 modules are still causing a problem.
- If node101 and node103 crash then their memory will be reverted as well.
Meeting with Ant about Go-Geo! metadata creation:
- Ant brought a paper document making various points about the interface and ambiguities in metadata creation process with the Go-Geo! Metadata Creator
- We longed for the integration of Grade and the Go-Geo! Metadata Creator
- Organisations getting data often specify too detialed a use, so that the licence restricts further dissemination under that licence.
- Meet again within the week to talk about this and Interreg IVC
- Actions
- Andy to contact Bex to find out progress on publishing.
- Ant to contact Bex about our plans to offer incentives for metadata creation.
- Browsing
Debugging Grids:
- Still having problems processing José's data - Geomorphometrics Use Case:
- OutOfMemoryError handling still not working correctly since code refactoring to expose the handling at a higher level.
SoG
- Talked to Adam regarding UK Human Population Census migration statistics and the counting of communal establishment populations.
- In his talk at the CSAP meeting 2007-10-30 Adam noted that smaller numbers of males migrate than females in age groups 15-24.
- Part of this is that more females carry on to further education which tend to involve migration.
- Another possible cause is that more young men end up in prison.
- Adam is going to investigate how the counting of communal establishment migration is done.
- If someone enters a communal establishment from a household then returns to that household in the same usual residence, then have they migrated, or does that depend on how long they were in the communal establishment?
- What are the details of the migration semantics?
OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- Evaluating SAW-GEO
- MoSeS
- Beowulf processing check:
- Received and written result with 117 HSARDataRecords and 77 ISARDataRecords for 38UCGP0014 with fitness 10.73956585816401059972
- Sending input 145621 with ZoneCode 38UCGR0002 to 9
- Debugging Grids:
- Still having problems processing José's data - Geomorphometrics Use Case:
- OutOfMemoryError handling still not working correctly since code refactoring to expose the handling at a higher level.
![[Validate RDF]](http://www.geog.leeds.ac.uk/people/a.turner/images/rdf_w3c_button.gif "W3C RDF Validation")
![[Validate CSS]](http://www.geog.leeds.ac.uk/people/a.turner/images/vcss.gif "W3C CSS Validation")
![[An image of Andy Turner]](http://www.geog.leeds.ac.uk/people/a.turner/a.turner.png)
