-
- Browsing
- Preparation for next week:
- GEOG5060
- A quick look on the portal
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 1: Done Chunk ( 14, 18 )
- Accepted Sorel Reisman's invitation onto the steering committee of "The IEEE Transactions on Computational Intelligence and Artificial Intelligence in Games (T-CIAIG)"
-
- Browsing
- Completed The Samples of Anonymised Records Usage Audit: What do you use the SARs for?
- Meeting with Nanlin Jin about Genetic Algorithms for Dynamic Land-use Optimization
- The abstract has been accepted to the WCCI 2008 Conference and the full paper is due 2008-03-15.
- We talked about the paper and about land management planning, practice and modelling.
- Together we will co-author a paper some day looking at land-use suitability, capability and economics in a dynamic complex system modelling way.
- For this paper I will remove myself from the list of authors, add an acknowledgement to RELU and myself (for time spent to improve the language).
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 1: Done Chunk ( 2, 51 )
- environment@leeds Meeting with Alan Haywood
- What do I want from this?
- What does Alan want from this?
- Details remain confidential for now.
- Invited Alan to join SciSpace in an effort to encourage further collaboration:
- Invite accepted and we are now both on this system :-)
- e-Science
- Browsing
- Attended compulsory Health and Safety event.
- Completed Fire Safety Training by noting all equipment and exits in building and having feedback session with Gordon Mitchell in his Fire Warden role.
- CAB/UCC Research Cluster Seminar 2008-02-27
- Colourful Wifi Network: Visualising identity in both real life and cyberspace
- Jung-Hua Liu
- http://fireant.itaiwan.net
- Users get location specific and personal blog/twitter.
- Mapping the difference between private and public wifi.
- Web 2.0 participation
- Various examples of bespoke community engagement involving wireless technology
- http://blog.pixnet.net
- MAC Address colouring.
- Red colours are happy colours in Taiwan :-)
- Users can change colours of a wifi access point somehow
- I didn't fully understand, but I think it is a bit like voting.
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 3: Done Chunk ( 20, 53 )
- I killed the run as the input data are wrong: (total time: 18,906 minutes 37 seconds). That's quite a contribution to global warming!
- Started run again on new data.
- e-Science
- Meeting with René Jordan
- A coffee time chat :-)
- I am reviewing a couple of papers next:
- One is for the 4th International Conference of e-Social Science
- The other I am listed as a co-author although I've not really contributed to it much yet.
- René is working on some mortality data for Phil Rees:
- I wondered what the source data were, so we started having a browse:
- Information organisation and eResearch
- I explained my way of blogging and distilling information and it's advantages.
- There is an overhead in getting this way of working going, and it can always be improved, but I argue that the benefits are potentially far greater than the effort required.
- Methodological issues abound in the development of new ways of working - new method.
- This is in part what eScience is all about.
- I will help René get set up to do this if it iss wanted.
- René is increasingly aware of the resistance to new ways of working amongst the established old school.
- It is good to be aware of this so as not to antagnoise the skeptics.
- René seemed impressed with how I manage to organise information about what I'm doing:
- I have been practising for a few years now and I am getting better and starting to realise the benefits.
- I will send René a link to this note. May the force be with you :-)
- Meeting with Steven Pickering
- We should talk more!
- http://inca.sdsc.edu/drupal/node/8
- Steven has been developing some monitoring software that can now query just about any device and visualise the data.
- A quick attempt to get it running on my PC didn't work, but that isn't what this meeting was about.
- Steven came to collect some data I pulled of GRADE yesterday:
- He intends to use the data for a splash screen for his monitoring software:
- Remote Inquiry and Visualisation of Eclectic Resources (RIVER).
- He will obide by the terms and conditions and give credit where it is due.
- We chatted about the next step of controlling devices and considered Sensor Web Enablement.
- Steven is in part working with Alan Haywood who I am meeting tomorrow to talk about environemnt@leeds :-)
- IAS is changing name to something with a C in it...
- Some links I found as we chatted:
- MoSeS
- Communication with Ann Harding from NATSEM
- I really like the NATSEM Website especially the Our People page.
- There are publications listed there, but I also found the following:
- Browsing
- Geomorphometrics
- Concerns of data IO...
- Progress on NextMap Cairngorm run:
- Iteration 3: Done Chunk ( 20, 21 )
- e-Science
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 3: Done Chunk ( 19, 47 )
- e-Science
- MoSeS
- Preparation for Technical Meeting tomorrow...
- Browsing
- Completed University of Leeds online fire safety training course available from:
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 3: Done Chunk ( 18, 9 )
- e-Science
- MoSeS
- Meeting with Mark
- Beowulf processing check:
- Received and written result with 129 HSARDataRecords and 0 ISARDataRecords for 00FYNL0034 with fitness 8.67498620022993387945
- This has not got any further, so my guess is that it has completed...
- NGS processing run:
- Did a run for 49860-50000 to test a new conficuration/setup
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 3: Done Chunk ( 17, 46 )
- HPC/WRG User Group 24 Meeting
- Based on the minutes of previous meetings, I won't need to take extensive notes :-)
- Documentation:
- Unfortunately, I arrived late and did not manage to access the wiki which had been working at the start of the meeting.
- Research Computing Data Storage (RCDS)
- Mark
- Hopefully the slides on the wiki...
- Replica data storage for backups
- Demo
- cd /rcds/home/%username%
- cd .zfs
- Check quota with df –h
- Storage Evolution Project
- Is Manuel or anyone from geography interested in buying more safe data store?
- If data in some files does not change much SRB is a sensible option for a data store.
- SRB is convenient but there is some overhead in IO, so it is not ideal for direct read/write from programs doing large amounts of IO. For these it is best to copy input to a large local temporary file store do the IO to/from this space putting results back into SRB at the end. Obviously this does not make full advantage of the massive data store SRB can make available, so if the program data requirements are of this scale then you have to take the io performance hit.
- Writing to /nobackup is reasonably slow, it is better on each node to write to the local scratch (not sure what the directoy path is).
- Referrence to faculty HPC meetings
- I could do with linking to my notes from the Faculty of Environment HPC Meeting...
- What should this User Group meeting be for?
- A forum for information about machine problems, how to improve our code e.g. to improve IO. How to factor HPC requirement into research grant proposals.
- Users do not need to know about the politics although they need to be aware that resources are limitted.
- e-Science
- http://www.myexperiment.org/users/164/friendships/395
- Meeting with Shiv Kaushal
- MoSeS
- Beowulf processing check:
- Received and written result with 129 HSARDataRecords and 0 ISARDataRecords for 00FYNL0034 with fitness 8.67498620022993387945
- I think this has finished...
- NGS processing run:
- Did a run for 49860-50000 to test a new conficuration/setup
- Browsing
- e-Science
- Telephone meeting with Chris Higgins
- Chris heading to OGF22 in Boston and is preparing for the OGC-OGF Workshop
- We talked through the current security arrangements on the MoSeS Use Case GeoLinking Service (GLS) implementations:
- Specifically dealing with the case where the GLS Client is configured to link data using Michael's GLS Server from 2001 Census attribute data served out via a GeoData Access Service (GDAS) hosted by MIMAS and the associated boundary data served out by a Web Feature Service (WFS) hosted by EDINA UKBorders.
- There is a login and registration system in place for the GLS Clients hosted on the NCeSS portal and the MoSeS portal.
- For the NCeSS portal there is more control for the MoSeS Worksites than the ESRC e-IP Worksite:
- The instance on the ESRC e-IP Worksite should probably be removed and the portlet should not be available as a tool for other worksites to add.
- On both portals we are trusting the systems administrators implicitly.
- There is no specific security in place to constrain use of Michael's GLS Server which is tied by IP address to work with the GDAS and WFS.
- The URLs for the GDAS and WFS are not publicly available and this is the only real security?
- Not really, these services will only work with Michael's GLS Server, so is it the URL of that that we need to be careful with?
- Maybe in turn it is tied to MoSeS GLS Client on geo-s12 for UK Census data GeoLinking?
- The security is not ideal, indeed not really sure how good it is, but things are like this because we are trying to understand what to do and for development purposes.
- Anyway, we can argue that no major risks have been taken.
- I wonder if someone guessing the right URL and request could in theory access this albeit small subset of Census Aggregate Statistics, census data nonetheless which is worng and for which we would be at fault.
- The security in place is not satisfactory and this will need to be improved.
- We talked though security options for GLS
- We are primarily concerned with data security.
- I mentioned that I had talked with Junaid about this.
- There are various options:
- We could tie the GDAS WFS and GLS Server and Client down so that they will only work with each other specifically:
- This goes against the grain of having any GLS client working with any GLS Server:
- This was perhaps only a requirement for developing the GLS Specification as part of the OGC GLS Interoperability Experiment.
- I can still appreciate that it would be useful to be able to use different Clients and Servers.
- Maybe we could run seperate tomcat instances or something for this, or use a specific port number?
- This could be a reasonably quick fix.
- What is probably best is that a security layer is added to the data ends (i.e. around the GDAS and WFS):
- This would require that a request for data demands some security is gone through (i.e. a Authorisation, Authentication and Access Service/mechanism is required).
- Either both the GLS client or the GLS Client and Server then have to handle authorisation requests and get user information to authenticate prior to data being linked and made available.
- Chris suggested that the GLS Server handles communication with the GDAS/WFS using security in WPS.
- This should work fine so long as the Server and Client communicate so that user provided certificates or other authentication details (provided to the client or client environment) enable access.
- Chris mentioned that there may be an alternative using Grid Security Infrastructure (GSI).
- Michael's GLS Server is running on a server machine that is used for various development and is not a production server with known service level agreement.
- For a more production service set up we could do with hosting the GLS Server on a more dedicated resource.
- A plan which is in the pipeline as part of the NCeSS e-Infrastructure for the social sciences project.
- We did not really consider Denial of Service attacks involving repeated authentication requests.
- I am to put all this in an email and send it to Chris, Junaid, Wei and Xiaobo.
- Chris has reopened a dialog with NGS
- There is an upcoming telecon about this
- There is work in the pipeline to get OS MasterMap on the NGS served out via WFS:
- Chris is dealing with Andy Richards, Neil Geddes and Gordon Brown from NGS/RAL to this end.
- I informed Chris that there had been a change in the deliverables of the NCeSS e-Infrastructure for the Social Sciences Work Package 4.1
- We are now looking to implement a secure GLS Server and make this available on the NCeSS Portal and perhaps on the NGS
- Chris wants me to refer him to some documentation on this.
- Andy Actions
- Send Security Email to Xiaobo, Junaid, Wei, Chris and Michael.
- Refer Chris to documentation on changes and details of NCeSS e-Infrastructure for the Social Science Work Package 4.1 deliverables.
- Provided a copy of the Land Registry data I scraped last year to Crispin Cooper.
- CSAP Seminar
- Attendees 15
- "Visualising 4d census interaction data (and some ideas on networks and the property market)"
- Crispin Cooper
- Great migration work :-)
- Can Crispin repeat, but using D Guo's method of organising data somost interactions are on the y = -x line?
- What happened to the interaction data in Lambeth?
- Houseprice data
- Phil outlined the deductive modelling approach of MIGMOD
- I mentioned the Agent Model work of Belinda Wu on MoSeS and for her PhD
- Phil says this is now asking what can do a better job, an agent based model, or a deductive model.
- Tony Champion has done related work on migration.
- Liam Birkett
- Research placement undergraduate student geo5lb.
- I will email Liam to ask about seeting him up a CCG web page.
- envirnoment@leeds Meeting 2007-02-19
- Geomorphometrics
- Progress on NextMap Cairngorms run:
- Iteration 3 : Done Chunk ( 16, 37 )
- e-Science
- Browsing
- Registered for http://lssi.leeds.ac.uk/events/50
- Geomorphometrics
- Progress on NextMap Cairngorm runs:
- Iteration 3 : Done Chunk ( 16, 2 )
- e-Science
- MoSeS
- Beowulf processing check:
- For some reason processing 69120-70000 halted prematurely. Decided not to investigate, but to set running again...
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Done Chunk ( 5, 58 )
Cannot read file D:\Work\Projects\Geomorphometrics\Workspace\data\NextMap\Workspace\6_0 in uk.ac.leeds.ccg.andyt.grids.core.Grid2DSquareCellDouble.loadObject(File)
java.lang.Error
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.loadObject(AbstractGrid2DSquareCell.java:5089)
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.loadIntoCacheChunk(AbstractGrid2DSquareCell.java:4950)
at uk.ac.leeds.ccg.andyt.grids.core.Grid2DSquareCellDouble.getGrid2DSquareCellDoubleChunk(Grid2DSquareCellDouble.java:1508)
at uk.ac.leeds.ccg.andyt.grids.core.Grid2DSquareCellDouble.getGrid2DSquareCellDoubleChunk(Grid2DSquareCellDouble.java:1459)
at uk.ac.leeds.ccg.andyt.grids.core.Grid2DSquareCellDouble.getGrid2DSquareCellDoubleChunk(Grid2DSquareCellDouble.java:1418)
at uk.ac.leeds.ccg.andyt.grids.process.Grid2DSquareCellProcessorDEM.getMetrics1(Grid2DSquareCellProcessorDEM.java:2813)
at uk.ac.leeds.ccg.andyt.grids.process.Grid2DSquareCellProcessorDEM.getMetrics1(Grid2DSquareCellProcessorDEM.java:2555)
at uk.ac.leeds.ccg.andyt.projects.geomorphometrics.Test.do_Metrics1(Test.java:485)
at uk.ac.leeds.ccg.andyt.projects.geomorphometrics.Test.run1(Test.java:280)
at uk.ac.leeds.ccg.andyt.projects.geomorphometrics.Test.run(Test.java:232)
at uk.ac.leeds.ccg.andyt.projects.geomorphometrics.Test.runOnPC(Test.java:123)
at uk.ac.leeds.ccg.andyt.projects.geomorphometrics.Test.main(Test.java:98)
BUILD SUCCESSFUL (total time: 716 minutes 21 seconds)
- Perhaps input data not completely loaded and swapped, or file locked? Initilisation from disc failure is worrying as grid should have been completely loaded and swapped to disc initially!
- Trying again, but loading from ascii file...
- e-Science
- MoSeS
- Beowulf processing run:
- Set run going for 69120-70000.
- Requested node103 reboot.
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- java.lang.StackOverflowError
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.writeToFileGrid2DSquareCellChunk(AbstractGrid2DSquareCell.java:3431)
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.writeToFileGrid2DSquareCellChunk(AbstractGrid2DSquareCell.java:3324)
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.swapToFileGrid2DSquareCellChunk(AbstractGrid2DSquareCell.java:4056)
at uk.ac.leeds.ccg.andyt.grids.core.AbstractGrid2DSquareCell.swapToFileGrid2DSquareCellChunk(AbstractGrid2DSquareCell.java:4012)
at uk.ac.leeds.ccg.andyt.grids.core.OutOfMemoryErrorHandler._SwapToFileGrid2DSquareCellChunk(OutOfMemoryErrorHandler.java:587)
at uk.ac.leeds.ccg.andyt.grids.core.OutOfMemoryErrorHandler.init_MemoryReserve(OutOfMemoryErrorHandler.java:212)
at uk.ac.leeds.ccg.andyt.grids.core.OutOfMemoryErrorHandler.init_MemoryReserve(OutOfMemoryErrorHandler.java:213)...
- What can make this occur?
- A memory leak.
- _MemoryReserve not initialised and cleared when it should be.
- _MemoryReserve not initialised to a sufficient size to allow for swapping operations.
- _MemoryReserve not being cleared before swapping operations called after OutOfMemoryError thrown.
- Insuffient stack:
- Can the stack size be made bigger?
- Consider handling StackOverflowErrors...
- Perhaps some other additional catch can be put in place to try and swap more data to disc in a single operation...
- The error is thrown initialising a grid... I made some changes and set things running again...
- e-Science
- NeSS
- Preparation for meeting tomorrow:
- Reading through documentation...
- MoSeS
- Beowulf processing:
- Could do with getting this going...
- NGS processing check:
- Still no output is being produced, but it seems the process is running :-(
- I'm going to kill the job and do some diagnosis...
- pbs-job.o56363
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- roughness_10.0.asc written out.
- e-Science
- MoSeS
- NGS processing check:
- Set run going for 69120-70000 but it may not work due to faults encountered last week.
- No output is being produced, but it seems the process is running.
- I'll should check this tomorrow in case I'm wasting resource allocation...
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorm run:
- Iteration 2 : Done Chunk ( 27, 11 )
- e-Science
- MoSeS
- What results in the Population Reconstruction for 2001 (Dynamic Model Population Initialisation) are yet to be produced? -
- MoSeSCheckAndFormatRun1.txt
- Problem in getCAS003Record(RecordID(218038)):
- This looks familiar... Do I need a special loader for Northern Ireland data?
- This is on-going...
- NGS processing check:
- Processing failing... I've tried various runs requesting different numbers of processes and I get something along the lines of the following:
- pbs-job.o54506
- I've emailed Shiv about this to ask his opinion. I think it may be that the default version of Java has changed and I need to moify the scripts to load Java1.5 rather than Java1.6.
- I'm struggling as http://www.grid-support.ac.uk/ is not working: The server at www.grid-support.ac.uk is taking too long to respond.
- The following command lists available modules:
- So I'm adding the java module for Java1.5 now, but the fault is persisting. I'm now trying to check if the fault is down to a change in the versions of the default mpi and other modules added...
- http://www.acet.rdg.ac.uk/projects/mpj/soft/readme.php
- Shiv Kaushal and Jason Lander reckon that the fault is that the myrinet network seems to have hit a limit of around 180 connections shared by all MPI jobs. I will be informed once progress is made...
- Browsing
- Joint EGC and RBPAM meeting to discuss funding proposals:
- People
- Attended by 17 people (some whose names I did not know). Here are a list of some:
- Others mentioned
- Alpine ecology decision support system:
- EGC Meeting
- Attended by 14/15 people (some whose names I did not know). Here are a list of some:
- ULPD should be automatically updated somehow...
- I suggested to Simon that his confidence in a measurement of biomass for plots/subplots is proportional to the length of plot boundary with nodata areas and the resolution of data collection.
- Simon's work is measuring above ground biomass only.
- Manuel working on estimating how representative plots are of larger regions...
- Simon comparing decade data 1980 v 1990s. I suggested doing all decades at say an annual time step. It seemed like this would be a lot of work...
- Pooling data resources
- Geomorphometrics
- Progress on NextMap run:
- Iteration 2 : Done Chunk ( 21, 21 )
- e-Science
- MoSeS
- What results in the Population Reconstruction for 2001 (Dynamic Model Population Initialisation) are yet to be produced? -
- NGS processing:
- Set run going for 69120-70000
- Browsing
- C&B Business meeting
- Attendees:
- After the meeting I had a chat with Harriet initially to find out her name, but then we talked about our common research interests :-)
- Harriet has a focus on the elderly population of Goole in East Riding and is conducting some survey to understand better the role of elderly can and do play in the communities of the town...
- Asked Hazel Parry to invite me to SciSpace social networking site for scientists.
- Thanks to Hazel I now have a profile at the following URL:
- Geomorphometrics
- Progress on NextMap run:
- Iteration 2 : Done Chunk ( 14, 56 )
- e-Science
- Staff Manager Meeting With Junaid Arshad
- Browsing
- CSAP Meeting
- People
- Attendees
- Others mentioned:
- Other details are confidential!
- Geomorphometrics
- Progress on NextMap Cairngorms run:
- Initialised result[45]
Done chunkRow 0
java.lang.OutOfMemoryError: Java heap space
BUILD SUCCESSFUL (total time: 550 minutes 15 seconds)
- This is not an identically run or result. The main difference was that the grid was loaded as a Grid rather than from the ASCII file.
- Probably should still consider to develop code to just provide the metric that is wanted, i.e. Roughness, rather than calculating and outputting lots which is what I am currently doing.
- Browsing
- Geomorphometrics
- Progress on NextMap Cairngorms run:
- java.io.IOException: There is not enough space on the disk
- I should develop code to just provide the metric that is wanted, i.e. Roughness, rather than calculating and outputting lots which is what I am currently doing.
- e-Science
- Browsing
- CCG
- Geomorphometrics
- Progress on NextMap Cairngorm runs:
- Iteration 1 writing out results...
- Steve Taylor has managed to inport the roughness_10.0.asc result from the first iteration into ArcGIS as a grid though for some reason the bottom part of the result is missing...
- e-Science
- MoSeS
- What results in the Population Reconstruction for 2001 (Dynamic Model Population Initialisation) are yet to be produced? -
- This will be on-going next week...