-
- Browsing
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- MoSeS
- Meeting with Shiv Kaushal and Paul Townend to discuss using NGS resources for MoSeS modelling work
- The population initialisation is approximately 200000 chunks of processing that can be done concurrently (in parallel).
- Plan to get MoSeS Population Initialisation code running on NGS resources in the next few weeks.
- Martin circulated a list of potential papers (titles and authors) for writing up over the next 3 months:
- The Dynamic Simulation Model - Computers, Environment and Urban Systems
- Population Reconstruction Model: a paper on the difficulties of making this work. ASAP journal
- A paper on the systems architecture: journal unknown
- A paper on how GRID (or Cloud) computing could seriously improve the speed at which the PRM could run. Currently we are generating results for 10,000 Output Areas per week on the Beowulf machine
- Beowulf processing check:
- Received and written result with 100 HSARDataRecords and 0 ISARDataRecords for 37UCGZ0002 with fitness 25.17448782301723478192
- Sending input 142896 with ZoneCode 37UCHA0016 to 7
-
- Browsing
- MASS meeting
- Kirk Harland organised a brainstorm on a problem he has with Spatial Interation Models:
- He is looking at the allocation of school places in leeds.
- If he uses a destination constrained model then all schools end up with about the same proportion empty.
- If he uses a destination unconstrained model then attractive schools end up with all the allcoation.
- In reality some schools are over subscribed and others are under-subscribed.
- I came up with an algorithm that might be worth investigating:
- Identify which schools are over-subscribed by doing destination unconstrained.
- For those over-subscribed use a destination constrained model to allocate to these.
- Take these schools and pupils out of the equation.
- Repeat 1 to 4 until no more over-subscribed schools.
- For the remaining schools run a destination unconstrained model.
- I wonder if 5 would ever be reached...
- Kirk plans to have a try of this and get back to us probably some time after Christmas.
- Probably a fully agent based model would be better.
- Belinda Wu asked for references on validation:
- Andy Evans asked if anyone knew about software for converting between Java Objects and XML.
- Later on I pointed him to
- CSAP meeting
- Migration/Mobility Themed Seminar
- Internal Migration of Great Britain at District Level
- Adam Dennett
- Useful measures:
- Churn, Turnover, Net Migration
- I wondered about all the reasons why smaller numbers of males than females aged 15 to 24 migrate.
- A communal establishment thing?
- Internal Migration of Britains Ethnic Groups
- Propensity to Migrate by Ethnic Group 2001 and 1991
- Paul Norman
- Migration analysis often uses NStates - e.g what social class is the person of after they have migrated:
- It is generally not known if this has changed.
- It is not captured in UK 2001 and 1991 Human Population Census Data.
- School Admissions Code 2007
- Kirk Harland
- Pupil Level Annual School Census
- Data about where people go to school and where they live and if they have moved schools or home in the last year.
- School Preference Data for Leeds
- Which schools are preferred and which are allocated.
- Policies on allocation explored.
- A random allocation is now a viable policy, but it results in increased average length of journeys compared with catchment constrained allocation.
- Pupil Level Annual School Census
- OGC Grid Collision, MoSeS, NCeSS e-Infrastructure for the Social Sciences
- NCeSS e-Infrastructure for the Social Sciences
- MoSeS
- Meeting
- Beowulf processing check:
- Received and written result with 134 HSARDataRecords and 0 ISARDataRecords for 36UGGW0014 with fitness 8.95659190885638817092
- Sending input 141971 with ZoneCode 36UGGY0003 to 20
-
- Browsing
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Completed 130001 - 140000
- Started 140001 - 150000
- Received and written result with 139 HSARDataRecords and 31 ISARDataRecords for 36UBGT0005 with fitness 3.58351777561255195970
- Sending input 140459 with ZoneCode 36UCGN0003 to 16
-
- Browsing
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 97 HSARDataRecords and 0 ISARDataRecords for 33UEHY0008 with fitness 14.99610647815670940781
- Sending input 135733 with ZoneCode 33UEHZ0010 to 4
-
- Geomorphometry
- Trying to understand results for José's DEM
- Looked for logic errors deep in the Grids code when it was the loading of the input data that was the problem.
- The big lesson is to check more thoroughly that you can input and output properly.
- A usual test for this is to import and export data and check that the source file and the exported file are the same.
- The test for difference should be done with an independent piece of software.
- The specific problem in this case was that my program was expecting an extra line in the header of the input specifying a NODATA_VALUE.
- I have not seen a specifications document for this file format, but I assume that the NODATA_VALUE line is an optional line in the header since of an ESRI Asciigrid format since ArcGIS handles these data fine.
- The upshot of all this is that I should regenerate the results I have produced for José. I have already informed him that I have identified the cause of the xshift.
- Every cloud has a silver lining and it was good to identify this problem. The bonus side effect is that I have a good idea how to do hillshading now.
- Indeed 0 is actually a No Data Value for José's DEM, so the input data was edited to add this extra line.
- SoG
- Impromtu meeting with Manuel Gloor
- He desperately and urgently wants to improve his web pages.
- He has talked with Mike Crabtree and knows to deal with David Appleyard about this.
- The problem is that David is away today.
- We checked to see if Manuel has an account on geogweb which he seems to, but cannot remember his password.
- I made him aware that Mike can create a SAMBA mapping to make it convenient for him to edit pages directly on the Web Server using his PC and Windows XP.
- He wants his web page to look more like Andy Evans
- I made him aware that style should be kept separate from content for accessibility reasons.
- I suggested Manuel get a meeting with David at the earliest opportunity to sort this out.
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 102 HSARDataRecords and 0 ISARDataRecords for 33UBHM0008 with fitness 18.17735212480396784947
- Sending input 134418 with ZoneCode 33UBHP0001 to 21
-
- Browsing
- Geomorphometry
- Trying to understand results for José's DEM
- Some results I am creating for José are interestingly wrong.
- It seems like some offset rounding error is coming into play.
- One of the results which should be the difference between the input and the output seems shifted.
- There should be no positive results, but there are!
- Loading the data into ArcGIS and taking the difference produces a result which looks like a hillshade.
- As a consequence I have an idea how to do hillshading.
- Subtract each value from another value offset by a number of rows and columns for all values in the grid.
- Error caused by conversion from row and column indexes to x and y coordinates or vice versa?
- That would be good to sort out!
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 123 HSARDataRecords and 0 ISARDataRecords for 32UDFS0006 with fitness 8.96006446125866303394
- Sending input 132793 with ZoneCode 32UDFT0008 to 2
- Role Analysis meeting to further specify tasks/job desription for security and tecchnical grid computing work on MoSeS and the NceSS e-Infrastructure for the Social Sciences project
- People
- I described the work being lined up for Junaid and what MoSeS is working on focussing on e-Infrastructure development and security issues.
- We agreed the following:
- Job Title: "Distributed Systems Researcher (Security Specialist)".
- Grade 6 Point 1
- Andy to redraft the Job Description and provide back to Louise for final checks.
- Andy to inform Junaid and Paul about progress.
- Draft job specification for work on MoSeS/NCeSS e-Infrastructure for the Social Sciences Projects 2
- Title and Grade:
- Distributed Systems Researcher (Security Specialist) Grade 6
- Summary
- As a distributed systems data access and security specialist you will join a research team to support MoSeS and NCESS-E-Inf.
- Primarily you will contribute to the development and implementation of security for MoSeS and NCESS-E-Inf.
- Project background
- MoSeS
- Modelling and Simulation for e-Social Science (MoSeS) is a research node of the National Centre for e-Social Science (NCeSS) funded for 3 years from July 2005.
- MoSeS is led by Mark Birkin and involves researchers/investigators in Geography, Computer Science, Health and Social Policy, and Transport at the University of Leeds.
- There are 3 main parts to the MoSeS work: Development of a demographic model of the UK; development of applications of the demographic model for business, health service planning and transportation research; Grid enabling and automation.
- NCESS-E-Inf
- The NCeSS e-Infrastructure for the Social Sciences Project (NCESS-E-Inf) involves MoSeS and other research nodes of NCeSS.
- NCESS-E-Inf is funded over 3 years from January 2007 and is divided into Work Packages.
- Work Package 4.1 is to develop an integrated end to end simulation workflow (from data inputs, simulation runs, outputs and analysis).
- Mike Daw from the University of Manchester is the overall project manager of this work. Mark Birkin and Andy Turner are project managing this work at the University of Leeds.
- Main Duties and Responsibilities
- Review security and develop appropriate solutions to:
- Improve the utility of available computational and data resources:
- This might involve providing security so that confidential data remains safe, and or looking at how to reschedule jobs so that those not involving confidential data can be processed on more open resources.
- Make information about the system, its data and use as openly available as allowed (under licence and user imposed constraints).
- Work with confidential and commercially valuable data under strict licence terms.
- Investigate secure ways of transferring data so that it cannot be readily intercepted by a third party.
- Support the integration of data sources by providing security such that it is as safe to integrate the data on various Grid resources as it is on a networked computer at the University of Leeds.
- Investigate and describe the risks associated with using different Grid resources with reference to particular datasets.
- Liaise with others to try and establish assurance that levels of security are in accordance with legal framework.
- Understand licences in place and consider if and what additional or modified licences are wanted.
- Contribute to the design of the system components that allow data to be collected and accessed in an automated way via a portal.
- Automate and document the system (configuration etc.)
- Attend and interact in meetings and support the dissemination of this work.
- Write up the results of your own research and contribute to research reports and publications.
- This will often be an iterative process, building in advice and guidance from others as appropriate.
- Present findings of research
- Prepare papers and make presentations with guidance and advice as appropriate.
- Work closely with others in the design and development of software solutions:
- Develop a team and team working skills.
- Ensure sufficient skills and knowledge overlap so the project delivery and future system development is not overly dependent on any individual team member.
- Personal Specification
- Essential
- A masters degree or PhD in Computational Geography, Informatics or related subject.
- Technical competence.
- Programming experience.
- Ability to work to deadlines and to a high standard.
- Excellent written and oral communication skills.
- Desirable
- Familiarity with GridSphere and other portal frameworks.
- Expertise in all aspects of computer security and Grid technology.
-
- Browsing
- Meeting with Anthony Beck to run through Go-Geo metadata creation and consider organising and taking on this work
- Agenda
- Create a metadata record together.
- Learn about the semantics and syntax of the metadata.
- Approximately how much time does it take to create a record?
- Can we produce 50 metadata records by the end of December?
- What next?
- Create a metadata record together.
- Step 1 Login to the system
- Step 2 Choose dataset
- Considered SRTM data.
- Decided on Houseprice data
- I scraped this data in June from http://www.houseprices.co.uk/
- Earlier scrapings were provided in house to:
- I thought it would be very useful data to collect and that it may be of use in MoSeS
- Step 3 Create the metadata record
- Learn about the semantics and syntax of the metadata.
- Some things still seem ambiguous despite the detail in the guidelines...
- Approximately how much time does it take to create a record?
- It depends on the attention to detail.
- 15 minutes to 1 hour.
- Can we produce 50 metadata records by the end of December?
- We're still relying on others to help with this and it is unknown if this will be forthcoming.
- What next?
- Ant to create another record and get on the system.
- Andy to write up some notes from the meeting.
- Arrange another meeting for about a weeks time when we know more about the metadata publishing process and have had more time to reflect on todays meeting.
- Other points
- We are a little worried about loading our colleagues with something that will come back to haunt them and us.
- It would be good to get back an XML file of the metadata.
- Maybe this happens after the publishing step.
- An XML file upload facility might help.
- OGC Grid Collision, MoSeS
- MoSeS
- Developing program to patch all Beowulf results together whilst checking for missing results:
- Hopefully patching all the results together is a one off job.
- What to do for the best probably depends on the completeness of results generated so far.
- The checks as below indicate that there is still considerable amount of processing left to do.
- For the initial subsets, where there are over say 500 Zones still to be processed, it is probably best to perform new parallel runs.
- Results should not be created for areas already processed:
- It is a waste of time and would make things difficult to produce the desired consistent single file result.
- Because 28 processes were used in parallel it is known that ( _StartRecordID + ( Number of processed zones ) + 28 ) is a good safe starting RecordID for further subset parallel runs.
- For example, to generate missing results in 40001 - 60000 RecordID range:
- Initiate a parallel process for _StartRecordID = 40001 + 3857 + 28, _endRecordID = 60000.
- Once majority of all results are generated then the results should be patched together into a single (ordered?) file:
- Ordering would be useful, but this could be done as a further processing step.
- It is useful as it would be easier to split into convenient chunks.
- I had been assuming that there were 175434 Output Areas, but only 174691 unique ZoneCodes are loaded in the OA to MSOA Look up table.
- Further investigation shows to expect the following total number of Output Areas:
- 223060 in UK
- 175434 Output Areas in England and Wales
- 165665 in England
- 9769 in Wales
- 5022 in Northern Ireland
- 42604 in Scotland
- Interestingly OAs are generally smaller in Scotland than elsewhere.
- Source: http://www.statistics.gov.uk/geography/census_geog.asp#oa
- Clearly this warrants further investigation as I am wanting to produce a UK dataset and it seems I am only processing (part of) England and Wales!
- Running Checks:
- 0 - 223060
- 87820 processed Zones
- 135240 Zones still to process!
- Approximately 39.4% complete...
- 0 - 30000
- 30000 processed Zones
- 0 Zones still to process!
- 30001 - 40000
- 10000 processed Zones
- 0 Zones still to process!
- 40001 - 60000
- 3857 processed Zones
- 16143 Zones still to process!
- 60001 - 70000
- 3082 processed Zones
- 6918 Zones still to process!
- 70001 - 80000
- 9994 processed Zones
- 6 Zones still to process!
- 80001 - 90000
- 9992 processed Zones
- 8 Zones still to process!
- 90001 - 100000
- 2240 processed Zones
- 7760 Zones still to process!
- 100001 - 110000
- 6873 processed Zones
- 3127 Zones still to process!
- 110001 - 120000
- 9997 processed Zones
- 3 Zones still to process!
- 120001 - 130000
- 1785 processed Zones
- 8215 Zones still to process!
- 130001 - 223060
- 0 processed Zones
- 93059 Zones still to process!
- There is quite some processing to be done still...
- Beowulf processing check:
- Received and written result with 127 HSARDataRecords and 0 ISARDataRecords for 31UGFU0011 with fitness 23.93337865742000644832
- Sending input 131372 with ZoneCode 31UGFY0001 to 26
- Browsing
- Changing a variable name based on the value of another variable in Java...
- I have sometimes thought this would be useful.
- I do not know of a shorthand way to do this, but a Map can be used as follows:
- All variable names can be put to the Map as keys with their respective values as values.
- To update a value one can pull the value from the Map using the variable name as the key and update it.
- In the example below the old mapping is replace (put over).
- To update a variable name, one can replaced (remove then put) the mapping in the Map.
- Before removing the mapping, a reference to the value must be kept in order that it can be put into the new mapping.
- Example code (available in HTML comment):
- //Initialise a HashMap and add two entries (variable names with values)
- HashMap _HashMap = new HashMap();
- String _VariableName;
- for ( int i = 0; i < 2; i ++ ) {
- _VariableName = new String( "v" + i );
- _HashMap.put( _VariableName , i );
- }
- //Print variable names and values
- System.out.println( "Initial variable names and values:" );
- Iterator _Iterator = _HashMap.keySet().iterator();
- Object key;
- int _Value;
- while ( _Iterator.hasNext() ) {
- key = _Iterator.next();
- _VariableName = ( String ) key;
- _Value = ( Integer ) _HashMap.get( key );
- System.out.println( _VariableName + " = " + _Value );
- }
- //Update a variable
- _VariableName = "v1";
- _HashMap.put( _VariableName, ( Integer ) _HashMap.get( _VariableName ) + 1 );
- //Update a variable name
- _VariableName = "v0";
- String _NewVariableName = "v2";
- Object value = _HashMap.get( _VariableName );
- _HashMap.remove( _VariableName );
- _HashMap.put( _NewVariableName, value );
- //Print variable names and values
- System.out.println( "Modified variable names and values:" );
- _Iterator = _HashMap.keySet().iterator();
- while ( _Iterator.hasNext() ) {
- key = _Iterator.next();
- _VariableName = ( String ) key;
- _Value = ( Integer ) _HashMap.get( key );
- System.out.println( _VariableName + " = " + _Value );
- }
- Example output:
- Initial variable names and values:
-
v1 = 1
- v0 = 0
- Modified variable names and values:
- v1 = 2
- v2 = 0
- When to put variables into such a Map?
- When it is useful to update valiable names.
- Not when this makes the code unnecessarily complex and execution heavy.
- Also consider putting into the mapping the type of the value in order that it can be readily cast upon retrieval.
- OGC Grid Collision, MoSeS
- MoSeS
- Developing program to patch all Beowulf results together whilst checking for missing results:
- Do we want all the results in a single file?
- Do we also want results in chunks?
- Probably...
- Local Authority District level divisions are probably best...
- Beowulf processing check:
- Received and written result with 124 HSARDataRecords and 0 ISARDataRecords for 31UBGR0011 with fitness 11.13947271782349304364
- Sending input 130269 with ZoneCode 31UCGG0015 to 10
- Browsing
- Geomorphometrics
- Modified getSlopeAndAspect(Grid2DSquareCellAbstract,double,double,double,Grid2DSquareCellDoubleFactory,boolean) in Grids so it now returns a Grid2DSquareCellAbstract for slope where the values are like a %slope.
- A high negative value indicates there is a steep up hill slope.
- A high positive value indicates there is a steep downhill slope.
- This is very close to the inverse of levelness a metrics1 metric.
- This should look nice draped over the DEM and visualised in 3D (2.5D).
- The only difference to before is that the values at the different scales can be directly compared and are effectively a %slope.
- To achieve this the metric at a given scale is normalised by the average distance and the total sum of all the distance weights used in the calculation.
- Thinking about combining slope and aspect measures:
- The sine of aspect can only be used to distinguish (contrast) two directions clearly
- The orthoganol directions are the same value irrespective of their aspect being in opposite directions.
- Is the aspect of more level areas of less interest than the aspect of steeper areas?
- Well, if the sun is orthoganol to the slope of a hill it is receiving more insolar radiation than flat areas.
- So, if the surface was a mirror then what would the observer see assuming the sun was effectively behind them?
- What about multiplying the sine of aspect with slope?
- Would this give a useful perception of landscape?
- Is this what I have known as hillshading?
- It is similar, but in a way it is assuming that the sun is higher in the sky than the angle of the greatest slope (i.e. that an area angled into the sun is not in the shaddow of another hill).
- What of more complex axis based metrics and measures of convex and concave...
- OGC Grid Collision, MoSeS
- NCeSS e-Infrastructure for the Social Sciences
- MoSeS
- Beowulf processing check:
- Received and written result with 99 HSARDataRecords and 0 ISARDataRecords for 29UBHH0004 with fitness 18.00841726094624011701
- Sending input 121810 with ZoneCode 29UBHJ0007 to 13
-
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x402fbc50, pid=4537, tid=1098562752
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode)
# Problematic frame:
# V [libjvm.so+0x2cdc50]
#
# An error report file with more information is saved as hs_err_pid4537.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
EXIT
mpi.MPIException: xdev.XDevException: java.nio.channels.ClosedChannelException
at mpi.Comm.send(Comm.java:378)
at mpi.Comm.Send(Comm.java:329)
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.run_OA(IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.java:343)
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.run(IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.java:114)
at uk.ac.leeds.ccg.andyt.projects.moses.process.IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.main(IndividualCensus_SWR_HSARHP_ISARCEP_Beowulf.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at runtime.daemon.JarClassLoader.invokeClass(JarClassLoader.java:113)
at runtime.daemon.ThreadedWrapper$1.run(ThreadedWrapper.java:240)
at java.lang.Thread.run(Thread.java:595)
Caused by: xdev.XDevException: java.nio.channels.ClosedChannelException
at xdev.niodev.NIODevice.isend(NIODevice.java:1377)
at xdev.niodev.NIODevice.send(NIODevice.java:1563)
at mpjdev.Comm.send(Comm.java:897)
at mpi.Comm.send(Comm.java:370)
... 11 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
at xdev.niodev.NIODevice.eagerSend(NIODevice.java:1693)
at xdev.niodev.NIODevice.isend(NIODevice.java:1372)
... 14 more
- Browsing
- OGC Grid Collision, MoSeS
- OGC
- e-Infrastructure for the Social Sciences Access Grid Meeting
- MoSeS
- Beowulf processing check:
- Received and written result with 99 HSARDataRecords and 0 ISARDataRecords for 29UBHH0004 with fitness 18.00841726094624011701
- Sending input 121810 with ZoneCode 29UBHJ0007 to 13
- Collaboration Land-Use Change
- Communications with Marcelo de C. C. Stabile
- Marcelo is a PhD student at the University of Sydney with a background in agronomy, (graduated in Brazil and with M.S. in Texas, working with Precision Agriculture).
- Starting a project on land use change due to climatic change in the hunter valley (a region north of Sydney).
- Intends to characterise and understand past land use change, and try to predict how uses may change in the future due to climatic change.
- Has very little programming experience, but wishes to learn.
- About data pre-processing, wouldn't there be a way to automate the task?
- I should let Marcelo know I am blogging this to help develop our collaboration.
- OGC Grid Collision, MoSeS
- Feedback to Paul about his OGF presentation slides, UK e-Social Science.
- MoSeS
- Beowulf processing check:
- Received and written result with 99 HSARDataRecords and 0 ISARDataRecords for 29UBHH0004 with fitness 18.00841726094624011701
- Sending input 121810 with ZoneCode 29UBHJ0007 to 13
- Collaboration Land-Use Change
- Communications with Marcelo de C. C. Stabile
- Edited email from Marcelo
- I just saw a presentation about "Predicting the Impact of Global Climatic Change on Land Use in Europe", which i found on the web.
- I would like to know if you have done any further research in fuzzy modelling, the presentation was very interesting and got my attention.
- I am starting a PhD and trying to figure out changes in Land use, based on forecasts of future climatic change (for a specific region), and while doing so, i want to incorporate other social and economic drivers in my predictions.
- Edited reply
- Sadly, I've not done any fuzzy modelling recently.
- What kinds of land use change do you intend to look at?
- Where are you working?
- Are you a programmer, or do you want to learn programming?
- What I did with my colleague Stan Openshaw (now retired) was the best we could manage at the time. The main restriction was the lack of data that is theoretically desirable for this work. Also, back in the late 1990's computational resources were far more limited. Additionally the software we used and developed is not up to todays standards. Anyway, we did produced some interesting results from which we could draw some conclusions to fuel debate. Various of the early publications as listed on my publications web page (http://www.geog.leeds.ac.uk/people/a.turner/publications/) are about this work. You should be able to access the full text of all but the book chapters. Please let me know if you come across any broken links.
- The last fuzzy modelling work I did was to do with combining physical indicators of land degradation risk. The fuzzy modelling was simplistic. In this like the previous work, the main lesson was that the data preprocessing was the hardest work.
- Thanks for getting back to me. I may be able to help further.
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 125 HSARDataRecords and 0 ISARDataRecords for 26UKFW0013 with fitness 6.98789077919178780287
- Sending input 121324 with ZoneCode 26UKFX0013 to 3
- Browsing
- Geomorphometry
- Adapted getSlopeAndAspect()
- Returns sine of aspect at N, NNE, ENE, E, ESE, SE and SSE angles, not just at N and E
- Normalised slope so that the value can be interpretted as a proportional slope.
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- 110001-120000 processing complete, but Nodes 102 and 103 are dead and Node101 is still running something.
- Started 120001-130000.
- Grids
- Working on memory management
- Exposing _Grid2DSquareCells and consolidating some memory handling code in OutOfMemoryHandler
- Abstracting Grid2DSquareCellProcessor and Grid2DSquareCellAbstract swapping methods...
- Geomorphometry
- Provided more results for José and set Slope and Aspect runs going.
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 112 HSARDataRecords and 0 ISARDataRecords for 26UEGA0016 with fitness 7.23260029557369278047
- Sending input 119561 with ZoneCode 26UEGB0007 to 25
- Browsing
- Grids
- Working on memory management
- Exposing _Grid2DSquareCells and consolidating some memory handling code in OutOfMemoryHandler
- Investigating commonalities between Grid2DSquareCellProcessor and Grid2DSquareCellAbstract swapping methods...
- Geomorphometry
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 143 HSARDataRecords and 0 ISARDataRecords for 24UNGL0003 with fitness 5.80584211697652196583
- Sending input 117780 with ZoneCode 24UNGP0001 to 7
- Browsing
- How to assess rendering quality?
- UoL
- SoG
- Meeting with José Muñoz-Rojas Morenés
- Transfered Grids results for scales 2 and 4.
- Plan for geomorphometrics work:
- Andy to complete processing of data for 8 and 16 scales.
- Andy to bundle code and give another demonstration of how it works so José can push other data through and play with resolutions etc.
- José to instigate collaboration with:
- Francisco Javier Tapiador
- A good friend, mainly working on the climate side of things now.
- Angel Alós
- This guy has left academia for now to work in the oil industry.
- Miguel Ángel Sanz Santos
- This guy works with Java and does some hard core geomorphometrics.
- Geomorphometry
- Grids and OutOfMemoryError Handling
- This continues the same item from yesterday.
- A concern is that it is not necessarily known how much memory is to be used in between calls to Grids methods where memory handling is resolved.
- There are two options:
- Explicitly handle OutOfMemoryErrors externally
- Two possibilities:
- Employ the memory handling in Grids to deal with any OutOfMemoryErrors
- A handle on _Grid2DSquareCells?
- After considering other options, this is probably best.
- Reserve memory externally and do some intricate OutOfMemoryError handling like in Grids.
- Ensure that on returning to external execution that there is sufficient memory to get back to Grids execution.
- Two possibilities:
- Set up some special reserve memory that can be allocated and deallocated on entry/exit.
- Know how much this is and pass this information into Grids and configure Grids to ensure it is available upon exit.
- This sounds ideal, but I have not thought of a good way to do this.
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 123 HSARDataRecords and 0 ISARDataRecords for 24UJGN0005 with fitness 3.50547349018015907537
- Sending input 116802 with ZoneCode 24UJGP0014 to 19
- Browsing
- At http://campus.leeds.ac.uk/kt/ the definition of knowledge transfer is one way (spin off). At http://campus.leeds.ac.uk/kt/faq.htm#1 it is more generally about transfer.
- This latter definition is better in todays world where researchers in the academic sector struggle to keep up with developments in IT and how they can be applied in their substantive fileds.
- For methods based research, Open Source and getting involved in Open Development initiatives is key to survival in the academic sector.
- PC Crash:
- It seemed like a hardware or power failure.
- It seems to have automatically rebooted fine.
- There were numerous processes and connections, but the load was not unusually heavy.
- I have contacted Geo-Support to ask if they can perform some diagnosis.
- Obviously this is a set back and I've lost unsaved work.
- Geomorphometry
- Grids and OutOfMemoryError Handling
- Thanks to Paul Townend who got back to me following a conversation about some memory problems I was having (summarised in the Geomorphometry item of my blog on 2007-09-14):
- More Background
- I first tried to develop a way to test if there was available memory when I was originally hunting for memory solutions to handling large raster data sets in Grids back in 2001.
- I did not get this test working satisfactorally then and began implementing a solution based on the try{}catch(){} construct of Java and the catching of OutOfMemoryErrors.
- Based on Paul's feedback I tried to develop a solution to the memory problems I was having and hacked away developing a program to utilise all available memory and identify what is the minimum amount of memory required to perform operations, for example, executing a for loop:
- I developed TestMemory.java to test for available memory and as a basis for calculating memory requirements:
- It will need much more work to evaluate the memory requirements for executing specific chunks of Java.
- While developing with Netbeans 5.5 I found it easiest to just use Debug mode as the program failed differently in normal run mode.
- As the program fails:
- _Runtime.maxMemory() = 66650112
- _Runtime.totalMemory() = 66650112
- _Runtime.freeMemory() = 0
- Is it reasonable to expect that a _TotalFreeMemory of 1024 is sufficient for an external program to call a method in Grids?
- Well, I assumed it was sufficient thinking that if this is not the case, then the amount can be increased as needed.
- So, one solution is to check to ensure there is enough memory to continue after collecting the result of a Grids API call before returning the result:
- To this end the following methods were added:
- Then I thought that more simply, _EnsureThereIsEnoughMemoryToContinue() could attempt to allocate a small amount of memory.
- I wonder which is faster?
- After reflecting, this is a far more elegant solution!
- The method can be moved to uk.ac.leeds.ccg.andyt.grids.utilities.OutOfMemoryErrorHandler
- It might swap a more preferred Grid2DSquareCellChunkAbstract.
- "Might" as this depends on what methods are soon to be called in the program execution.
- Actually for Chunk processing (going through all cell values (getCell() or setCell()) in each Grid2DSquareCellAbstract Grid2DSquareCellChunkAbstract by Grid2DSquareCellChunkAbstract) this should be quite good in that it will at least prevent a huge amount of swapping of the same Grid2DSquareCellChunkAbstract.
- Anyway, the public methods needed to be refactored to be something along the lines of the following (NB. The change is in the try{} block):
- public Object getSomething(
- Object[] args,
- boolean _HandleOutOfMemoryError ) {
- try {
- Object result = getSomething( Object[] args );
- _EnsureThereIsEnoughMemoryToContinue();
- return result;
- } catch ( OutOfMemoryError _OutOfMemoryError ) {
- if ( _HandleOutOfMemoryError ) {
- clearMemoryReserve();
- swapData(); // e.g. swapToFileGrid2DSquareCellsGrid2DSquareCellChunk(); or swapToFileGrid2DSquareCellsGrid2DSquareCellChunkExcept( ( ChunkID ) args[ 0 ] );
- initMemoryReserve();
- return getSomething(
- Object[] args,
- handleOutOfMemoryError );
- } else {
- }
- }
- }
OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 148 HSARDataRecords and 0 ISARDataRecords for 24UDGF0015 with fitness 1.23890855415128487191
- Sending input 115383 with ZoneCode 24UDGH0001 to 23
- Browsing
- OGC Grid Collision, MoSeS
- MoSeS
- Beowulf processing check:
- Received and written result with 118 HSARDataRecords and 0 ISARDataRecords for 24UBHR0006 with fitness 4.84865992629945081239
- Sending input 114297 with ZoneCode 24UBHS0009 to 12
- Draft job specification for work on MoSeS/NCeSS e-Infrastructure for the Social Sciences Projects
- Summary
- As a distributed systems data access and security specialist you will join a research team to support MoSeS and NCESS-E-Inf.
- Primarily you will contribute to the development and implementation of security for MoSeS and NCESS-E-Inf.
- Project background
- MoSeS
- Modelling and Simulation for e-Social Science (MoSeS) is a research node of the National Centre for e-Social Science (NCeSS) funded for 3 years from July 2005.
- MoSeS is led by Mark Birkin and involves researchers/investigators in Geography, Computer Science, Health and Social Policy, and Transport at the University of Leeds.
- There are 3 main parts to the MoSeS work: Development of a demographic model of the UK; development of applications of the demographic model for business, health service planning and transportation research; Grid enabling and automation.
- NCESS-E-Inf
- The NCeSS e-Infrastructure for the Social Sciences Project (NCESS-E-Inf) involves MoSeS and other research nodes of NCeSS.
- NCESS-E-Inf is funded over 3 years from January 2007 and is divided into Work Packages.
- Work Package 4.1 is to develop an integrated end to end simulation workflow (from data inputs, simulation runs, outputs and analysis).
- Mike Daw from the University of Manchester is the overall project manager of this work. Mark Birkin and Andy Turner are project managing this work at the University of Leeds.
- Main Duties and Responsibilities
- Review security and develop appropriate solutions to:
- Attempt to maximise the utility of available computational and data resources
- Make information about the system, its data and use as openly available as allowed (under licence and user imposed constraints).
- Automation and documention of the system (configuration etc.)
- Attendance and interaction in meetings and supporting the dissemination of this work.
- Working closely with others in the design and development of software solutions.
- Personal Specification
- Essential
- A masters degree or PhD in Computational Geography, Informatics or related subject.
- Technical competence.
- Programming experience.
- Ability to work to deadlines and to a high standard.
- Excellent written and oral communication skills.
- Desirable
- Familiarity with GridSphere and other portal frameworks.
- Expertise in all aspects of computer security and Grid technology.
- Browsing
- OGC Grid Collision, MoSeS
- GRADE Go-GEO collaboration
- Rebecca Seymore from EDINA contacted a group of GIS users at Leeds to ask for expressions of interest in creating AGMAP metadata (ISO19115 compliant) using the Go-Geo! 'Metadata Creator Tool' (accessed from the Home page from the "Tell others about your data" go. ).
- The offer is *£1000 for the creation of 50 metadata records*, recording datasets held at Leeds University.
- A Leeds University private local node (which is only visible to Leeds Athens users) has been created - you can see this if you login to the site and then click on "Advanced search". Metadata records can be published to your private local site only, or to the public Go-Geo! Network - whichever is perferred (or a combination of both depending on the sensitivities of the data).
- Cambridge, Sheffield, Kingston, Aberdeen and Edinburgh Universities are also part of this work and EDINA are looking for an additional institution.
- I've expressed interest (as has Anthony Beck) in being a contact and coordinator at Leeds University and taking on board this work.
- MoSeS
- OGC Grid Collision, MoSeS
- Meeting with Junaid about Security work in MoSeS and NCeSS e-Infrastructure for the social sciences project
- Junaid is interested in this work and awaits a job description which I am to draft.
- Browsing
- OGC Grid Collision, MoSeS
- SAW-GEO
- OGC
- MoSeS
- Beowulf processing check:
- 100000 to 110000 processing failed to complete having:
- Received and written result with 112 HSARDataRecords and 0 ISARDataRecords for 21UFGQ0006 with fitness 6.99482497386621214255
- Sending input 106900 with ZoneCode 21UFGQ0022 to 8
- Progress meeting
![[Validate RDF]](http://www.geog.leeds.ac.uk/people/a.turner/images/rdf_w3c_button.gif "W3C RDF Validation")
![[Validate CSS]](http://www.geog.leeds.ac.uk/people/a.turner/images/vcss.gif "W3C CSS Validation")
![[An image of Andy Turner]](http://www.geog.leeds.ac.uk/people/a.turner/a.turner.png)
