Welcome to TiddlyWiki created by Jeremy Ruston, Copyright © 2007 UnaMesa Association
Type the text for '(built-in shadow tiddler)'
!Why?
* because every time I bring out a graph to prove a point they say "ahh but thats on a 5 minute average"
* because James D. ~McCabe uses 30 second samples in his performance testing
!How
The standard cricket RRD structure is designed for data collected at 5 minute intervals. To collect cricket data on non-standard intervals a new RRD structure and target types to use it must be defined.
!!RRA Definition
This is the Round Robin Database (RRD) Round Robin Archive (RRA) definition. It controls how cricket stores its data. Put this in the RRA section of one of your //Defaults// files. I add it to the top level //Defaults// file.
In these RRA definitions the important values, in this case, are the last two numbers. The last number is the total number of data points in the series and the second last is the number of data points that are consolidated to produce a point in this array, see [[RRDTool]].
In the first RRA definition the important change is the total number of data points. This RRA contains the unaltered captured data and it must be large enough to store the required number of data points, in this case two days worth (50 hours * 60 mins * 60 secs / 30 secs).
The subsequent ~RRAs contain consolidated data and are either averages or maximums of that data. The total number of data points in these ~RRAs does not need to be changed since we do not wish to change the length of time they represent. The number of base data points used to produce a point of data in this array does however need to be altered. In the second RRA for example the number of base points consolidated to produce each point in the series is 60 (30 mins * 60 secs / 30 secs)
{{{
# one point every 30 seconds for 50 hours
rra 30secAve AVERAGE:0.5:1:3760
# one point every 30 minutes for 12 days, both average and maximum
rra 30minAves30 AVERAGE:0.5:60:600
rra 30minMaxs30 MAX:0.5:60:600
# one point every 2 hours for 50 days, both average and maximum
rra 2hrAves30 AVERAGE:0.5:240:600
rra 2hrMaxs30 MAX:0.5:240:600
# one point every day for 600 days, both average and maximum
rra 1dayAves30 AVERAGE:0.5:2880:600
rra 1dayMaxs30 MAX:0.5:2880:600
}}}
This new RRD will use about 10 times the storage and use about 10 times the performance to collect of the standard cricket RRD, so it is probably wise to use it sparingly.
!!Target Definition
First define a new target type using the new RRA definition we just added. This can be placed in any //Defaults// in or above the 30 second collection subtree, I put it in my top level //Defaults// file.
{{{
targetType standard-interface-30sec
ds = "inInOctets, inOutOctets"
view = "Octets: inInOctets ifOutOctets"
rra = "30secAve, 30minAves30, 30minMaxs30, 2hrAves30, 2hrMaxs30, 1dayAves30, 1dayMaxs30"
}}}
Unfortunately the poll interval is a target dictionary property. I put anything I am collecting faster than 5 minutes in its own configuration subtree to simplify the //subtree-sets// definitions and add this entry to the top level //Defaults// file in that subtree.
{{{
Target --default--
target-type = standard-interface-30sec
rrd-poll-interval = 30
}}}
!!Cron
Cron does not allow execution of programs at intervals of less than one minute. Cron however allows you to execute multiple commands on the same line. So the workaround is to run the cricket collector twice each minute using sleep to delay the second run by 30 seconds.
Here is an example crontab...
{{{
* * * * * $HOME/cricket/collect-subtrees high-resolution
* * * * * sleep 30; $HOME/cricket/collect-subtrees high-resolution
}}}
All the 30 second targets are in a subtree set called high-resolution.
This site is a vehicle for my thoughts and experiences in network management and particularly in network management using opensource tools. Check the sections under [[Network Management Model]] for information on network management tools and [[My Projects]] for stuff that I have written.
I used [[TiddlyWiki|http://www.tiddlywiki.com]] to construct this site.
Type the text for 'New Tiddler'
Bind 9.5 introduced a new and extensive set of statistics counters. It also introduced an XML statistics channel that could be accessed by http rather than having to get statistics from a local file.
I have written a perl script called bind9_5_counters.pl to extract values from the BIND 9.5 DNS server XML statistics channel. It produces reports in [[Cricket]] [[MRTG]] and [[Cacti]] format, however I have only tested it with [[Cricket]]. If you are interested you can read the [[man page|projects/bind9_5_counters/bind9_5_counters.html]].
Now updated for Bind 9.6.1! You can download it here [[bind9_5_counters|projects/?project=bind9_5_counters]].
{{center{
[img[Example Cricket Graph|images/dns.1.small.png][images/dns.1.png]]
Example Cricket Graph
}}}
~BigBrother is a fault management system with a simple philosophy, green is good, red is bad.
| [img[Sample BigBrother Screen|images/bb.jpg]] |
| Sample ~BigBrother Screen |
| borderless |k
There are a large number of add-on modules and extensions available. If you are using ~BigBrother you must look at bbgen, it improves performance dramatically and adds extra functionality. ~BigBrother also has a graphing extension larrd which can produce graphs from values such as response times and disk space.
~BigBrother has been bought by Quest Software, they still offer a free version however development seems to have stalled.
The [[Hobbit]] is a fork of the ~BigBrother system developed by the team that produced bbgen. It seems to be the logical successor to ~BigBrother.
You can get ~BigBrother from http://www.bb4.org/ just look for the link to BTF.
<script>
if (!window.story) window.story=window;
var url=store.getTiddlerText("SiteUrl");
var title=story.findContainingTiddler(place).id.substr(7);
var permalink=encodeURIComponent(String.encodeTiddlyLink(title));
return "[img[Right click to Bookmark or Copy this tiddler Location|images/bookmarkicon.gif]["+url+"#"+permalink+"]]";
</script>
version.extensions.breadCrumbs = {major: 1, minor: 8, revision: 2, date: new Date("Oct 26, 2007")};
// show/hide display option (default is to SHOW breadcrumbs)
if (config.options.chkShowBreadcrumbs==undefined)
config.options.chkShowBreadcrumbs=true;
// REORDER breadcrumbs when visiting previously viewed tiddler (default is to TRIM breadcrumbs)
if (config.options.chkReorderBreadcrumbs==undefined)
config.options.chkReorderBreadcrumbs=false;
// create default breadcrumbs display as needed (default is to CREATE)
if (config.options.chkCreateDefaultBreadcrumbs==undefined)
config.options.chkCreateDefaultBreadcrumbs=true;
// show breadcrumbs for 'startup' tiddlers (default is FALSE = only show crumbs for tiddlers opened after startup)
if (config.options.chkShowStartupBreadcrumbs==undefined)
config.options.chkShowStartupBreadcrumbs=false;
config.macros.breadcrumbs = {
crumbs: [], // the list of current breadcrumbs
handler: function(place,macroName,params,wikifier,paramString,tiddler) {
var area=createTiddlyElement(place,"span",null,"breadCrumbs",null);
area.setAttribute("homeSep",params[0]?params[0]:this.homeSeparator); // custom home separator
area.setAttribute("crumbSep",params[1]?params[1]:this.crumbSeparator); // custom crumb separator
this.render(area);
},
add: function (title) { // ELS: changed from passing event, "e", to passing tiddler title
var thisCrumb = "[[" + title + "]]";
var ind = this.crumbs.find(thisCrumb);
if(ind === null)
this.crumbs.push(thisCrumb);
else if (config.options.chkReorderBreadcrumbs)
this.crumbs.push(this.crumbs.splice(ind,1)[0]); // reorder crumbs
else
this.crumbs=this.crumbs.slice(0,ind+1); // trim crumbs
this.refresh();
return false;
},
getAreas: function() {
var crumbAreas=[];
// find all DIVs with classname=="breadCrumbs"
// Note: use try/catch to avoid "Bad NPObject as private data" fatal error caused when
// embedded QuickTime player element is accessed by hasClass() function.
var all=document.getElementsByTagName("*");
for (var i=0; i<all.length; i++)
try{ if (hasClass(all[i],"breadCrumbs")) crumbAreas.push(all[i]); } catch(e) {;}
// find single DIV w/fixed ID (backward compatibility)
var byID=document.getElementById("breadCrumbs")
if (byID && !hasClass(byID,"breadCrumbs")) crumbAreas.push(byID);
if (!crumbAreas.length && config.options.chkCreateDefaultBreadcrumbs) { // no existing crumbs display areas... create one...
var defaultArea = createTiddlyElement(null,"span",null,"breadCrumbs",null);
defaultArea.style.display= "none";
var targetArea= document.getElementById("tiddlerDisplay");
targetArea.parentNode.insertBefore(defaultArea,targetArea);
crumbAreas.push(defaultArea);
}
return crumbAreas;
},
refresh: function() {
var crumbAreas=this.getAreas();
for (var i=0; i<crumbAreas.length; i++) {
crumbAreas[i].style.display = config.options.chkShowBreadcrumbs?"block":"none";
removeChildren(crumbAreas[i]);
this.render(crumbAreas[i]);
}
},
render: function(here) {
createTiddlyButton(here,"Home",null,this.home,"tiddlyLink tiddlyLinkExisting");
for (c=0; c<this.crumbs.length; c++)
if (!store.tiddlerExists(this.crumbs[c].replace(/\[\[/,'').replace(/\]\]/,'')))
this.crumbs.splice(c,1); // remove non-existing tiddler from crumbs
var homeSep=here.getAttribute("homeSep"); if (!homeSep) homeSep=this.homeSeparator;
var crumbSep=here.getAttribute("crumbSep"); if (!crumbSep) crumbSep=this.crumbSeparator;
wikify(homeSep+this.crumbs.join(crumbSep),here);
},
home: function() {
story.closeAllTiddlers();
restart();
config.macros.breadcrumbs.crumbs = [];
var crumbAreas=config.macros.breadcrumbs.getAreas();
for (var i=0; i<crumbAreas.length; i++) crumbAreas[i].style.display = "none";
return false;
}
};
if (config.macros.breadcrumbs.homeSeparator==undefined) // note: not a cookie
config.macros.breadcrumbs.homeSeparator=" | ";
if (config.macros.breadcrumbs.crumbSeparator==undefined) // note: not a cookie
config.macros.breadcrumbs.crumbSeparator=" > ";
config.commands.previousTiddler = {
text: 'back',
tooltip: 'view the previous tiddler',
hideReadOnly: false,
dateFormat: 'DDD, MMM DDth YYYY hh:0mm:0ss',
handler: function(event,src,title) {
var here=story.findContainingTiddler(src); if (!here) return;
var crumbs=config.macros.breadcrumbs.crumbs;
if (crumbs.length>1) {
var crumb=crumbs[crumbs.length-2].replace(/\[\[/,'').replace(/\]\]/,'');
story.displayTiddler(here,crumb);
}
else
config.macros.breadcrumbs.home();
return false;
}
};
config.macros.previousTiddler= {
label: 'back',
prompt: 'view the previous tiddler',
handler: function(place,macroName,params,wikifier,paramString,tiddler) {
var label=params.shift(); if (!label) label=this.label;
var prompt=params.shift(); if (!prompt) prompt=this.prompt;
createTiddlyButton(place,label,prompt,function() {
var crumbs=config.macros.breadcrumbs.crumbs;
if (crumbs.length>1) {
var crumb=crumbs[crumbs.length-2].replace(/\[\[/,'').replace(/\]\]/,'');
story.displayTiddler(place,crumb);
}
else
config.macros.breadcrumbs.home();
});
}
}
// hijack story.displayTiddler() so crumbs can be refreshed when a tiddler is displayed
if (Story.prototype.breadCrumbs_coreDisplayTiddler==undefined)
Story.prototype.breadCrumbs_coreDisplayTiddler=Story.prototype.displayTiddler;
Story.prototype.displayTiddler = function(srcElement,title,template,animate,slowly)
{
this.breadCrumbs_coreDisplayTiddler.apply(this,arguments);
// if not displaying tiddler during document startup, then add it to the breadcrumbs
// note: 'startingUp' flag is a global, set/reset by the core init() function
if (!startingUp || config.options.chkShowStartupBreadcrumbs) config.macros.breadcrumbs.add(title);
}
// hijack store.removeTiddler() so crumbs can be refreshed when a tiddler is deleted
if (TiddlyWiki.prototype.breadCrumbs_coreRemoveTiddler==undefined)
TiddlyWiki.prototype.breadCrumbs_coreRemoveTiddler=TiddlyWiki.prototype.removeTiddler;
TiddlyWiki.prototype.removeTiddler= function(title)
{
this.breadCrumbs_coreRemoveTiddler.apply(this,arguments);
config.macros.breadcrumbs.refresh();
}
Graphs and data collection in Cacti are base on templates written in XML. Once a template is defined for a data source the device configuration is done from a web based form. Device configuration is stored in an SQL database. Cacti is based on [[RRDTool]]. Cacti supports individualised views of the data and the data an individual can view can be restricted. Navigation in Cacti is by a menu tree that is contained in the left pane of the main window and there is a preview mode which shows a reduced view of all graphs.
| [img[Sample Cacti Screen|images/cacti.png]] |
| Sample Cacti Screen (from cacti.org) |
|borderless|k
You can get cacti from http://www.cacti.net.
Recently I have been working with SNMPTT and SEC. SNMPTT (SNMP Trap Translator) takes SNMP traps received by snmptrapd and converts them into plain language. I have been having fun with some Cisco traps and I found that my problem is that Cisco doesn't always follow the relevant MIBS in regard to trap arguments.
I found that the answer is to have two versions of the trap description and then use a match directive to differentiate between them (in this case the number of arguments)...
># Cisco Version
>EVENT linkUp .1.3.6.1.6.3.1.1.5.4 "Status Events" Normal
>FORMAT Link up on interface $2 ($1), media $3, reason $4
>MATCH $#: 4
>#
># Standard Version
>EVENT linkUp .1.3.6.1.6.3.1.1.5.4 "Status Events" Normal
>FORMAT Link up on interface $1. Admin state: $2. Operational state: $3
>MATCH $#: 3
In the case of this trap you can make IOS send a standard trap with the command...
>snmp-server trap link ietf
But there isn't a command to do this for some of the other traps and I prefer the Cisco arguments for the linkUp/Down traps.
The traps I have found Cisco devices to send non-standard arguments for are linkUp, linkDown and all the ~IEEEdot11 MIB traps.
Collect store and archive inventory and configuration of network devices.
Counters.pl collects the CLI counters from network devices and produces custom reports. The reports can be viewed via a web site and/or sent via email. There is a flexible configuration language in which reports can be specified. The format and style of the html reports are controlled by Template Toolkit templates and css.
Why would you want to collect CLI counters? The CLI counters can be reset independently of the SNMP counters and when collected weekly can highlight issues such as connection errors that you will not see looking at graphs produced from SNMP counters. They also contain information that is not available via SNMP. I run counters.pl early every Friday morning and review the results before lunch on that day.
Currently only Cisco network equipment is supported.
Now updated to with index fancification! (Version1.05)!
You can download it here [[counters.pl|projects/?project=counters]].
{{center{
[img[Example HTML Report|images/counters.small.png]]
Example HTML Report
}}}
Produces statistics of POP and IMAP use on a Courier server by processin the logs.
| [img[Sample Couriergraph Graph|images/couriergraph.png]] |
| Sample Couriergraph Graph |
|borderless|k
You can get Couriergraph from http://www.arschkrebs.de/postfix/couriergraph/.
Cricket is a performance management system based on RRDTool. Its configuration has a directory hierarchy which controls the structure of the navigation system. Default graph and data properties of lower level entries are inherited from higher level entries. Cricket navigation uses a hierarchy of descriptive text tables that lead to leaf pages that contain graphs spanning several time periods.
| [img[Sample Cricket Page|images/cricket.jpg]] |
| Sample Cricket Page |
| borderless |k
Cricket's presentation is constrained to a small selection of views, however this should be sufficient for most uses. An external graphing program such as [[ddraw]] or [[RRDgrapher]] could be used to present extra views of the data collected by Cricket. Cricket graphs can display an arbitrary number of data sequences in contrast to [[MRTG]] which classically displayed two data sequences per graph and many [[RRDTool]] graphing features, such as stacked graphs, can be used.
Cricket has a novel feature called 'instance mapping' that allows SNMP instances to be referenced by names stored in the SNMP tables themselves. This protects Cricket against SNMP instances that change index.
Development of Cricket has stalled however it is a mature and well developed package.
You can get cricket from http://cricket.sourceforge.net.
[[About]]
[[My Projects]]
[[My Papers]]
[[Network Management Model]]
[[Resources]]
[[Event Handling]]
[[Visitors]]
Syslog and SNMP traps are a common source of Network Management events.
config.macros.timeline.handler = function(place,macroName,params)
{
var field = params[0] ? params[0] : "modified";
var tiddlers = store.reverseLookup("tags","excludeTimeline",false,field);
var lastDay = "";
var last = params[1] ? tiddlers.length-Math.min(tiddlers.length,parseInt(params[1])) : 0;
for(var t=tiddlers.length-1; t>=last; t--)
{
var tiddler = tiddlers[t];
var theDay = tiddler[field].convertToLocalYYYYMMDDHHMM().substr(0,8);
if(theDay != lastDay)
{
var theDateList = document.createElement("ul");
place.appendChild(theDateList);
createTiddlyElement(theDateList,"li",null,"listTitle",tiddler[field].formatString(this.dateFormat));
lastDay = theDay;
}
var theDateListItem = createTiddlyElement(theDateList,"li",null,"listLink");
theDateListItem.appendChild(createTiddlyLink(place,tiddler.title,true));
}
}
Managing network problems. The major role of fault management systems is to detect network problems and report them. A more advanced system may attempt to rectify the problem.
| [img[Basic Fault Management System|images/fault-management-system.png]] |
| Basic Fault Management System |
| borderless |k
!Fault Management Systems
* [[Mon]] is an older system that still lends itself to smaller intallation or specialised uses
* [[BigBrother]] is a simple system with an intuitive human interface that can handle quite a good sized site
* [[Nagios]] is probably the current premier opensource fault management system
!Alerting
* [[QuickPage]] is a system for messages to pagers
* [[SMS Server Tools]] is a system for sending and receiving SMS messages
* [[SMSLink]] is another SMS sending / receiving system
* [[gnokii]] is an API and tools for talking to mobile phones
!Event Handling
*[[This is a whole section on event handling|Event Handling]]
/***
|''Name:''|ForEachTiddlerPlugin|
|''Version:''|1.0.8 (2007-04-12)|
|''Source:''|http://tiddlywiki.abego-software.de/#ForEachTiddlerPlugin|
|''Author:''|UdoBorkowski (ub [at] abego-software [dot] de)|
|''Licence:''|[[BSD open source license (abego Software)|http://www.abego-software.de/legal/apl-v10.html]]|
|''Copyright:''|© 2005-2007 [[abego Software|http://www.abego-software.de]]|
|''TiddlyWiki:''|1.2.38+, 2.0|
|''Browser:''|Firefox 1.0.4+; Firefox 1.5; InternetExplorer 6.0|
!Description
Create customizable lists, tables etc. for your selections of tiddlers. Specify the tiddlers to include and their order through a powerful language.
''Syntax:''
|>|{{{<<}}}''forEachTiddler'' [''in'' //tiddlyWikiPath//] [''where'' //whereCondition//] [''sortBy'' //sortExpression// [''ascending'' //or// ''descending'']] [''script'' //scriptText//] [//action// [//actionParameters//]]{{{>>}}}|
|//tiddlyWikiPath//|The filepath to the TiddlyWiki the macro should work on. When missing the current TiddlyWiki is used.|
|//whereCondition//|(quoted) JavaScript boolean expression. May refer to the build-in variables {{{tiddler}}} and {{{context}}}.|
|//sortExpression//|(quoted) JavaScript expression returning "comparable" objects (using '{{{<}}}','{{{>}}}','{{{==}}}'. May refer to the build-in variables {{{tiddler}}} and {{{context}}}.|
|//scriptText//|(quoted) JavaScript text. Typically defines JavaScript functions that are called by the various JavaScript expressions (whereClause, sortClause, action arguments,...)|
|//action//|The action that should be performed on every selected tiddler, in the given order. By default the actions [[addToList|AddToListAction]] and [[write|WriteAction]] are supported. When no action is specified [[addToList|AddToListAction]] is used.|
|//actionParameters//|(action specific) parameters the action may refer while processing the tiddlers (see action descriptions for details). <<tiddler [[JavaScript in actionParameters]]>>|
|>|~~Syntax formatting: Keywords in ''bold'', optional parts in [...]. 'or' means that exactly one of the two alternatives must exist.~~|
See details see [[ForEachTiddlerMacro]] and [[ForEachTiddlerExamples]].
!Revision history
* v1.0.8 (2007-04-12)
** Adapted to latest TiddlyWiki 2.2 Beta importTiddlyWiki API (introduced with changeset 2004). TiddlyWiki 2.2 Beta builds prior to changeset 2004 are no longer supported (but TiddlyWiki 2.1 and earlier, of cause)
* v1.0.7 (2007-03-28)
** Also support "pre" formatted TiddlyWikis (introduced with TW 2.2) (when using "in" clause to work on external tiddlers)
* v1.0.6 (2006-09-16)
** Context provides "viewerTiddler", i.e. the tiddler used to view the macro. Most times this is equal to the "inTiddler", but when using the "tiddler" macro both may be different.
** Support "begin", "end" and "none" expressions in "write" action
* v1.0.5 (2006-02-05)
** Pass tiddler containing the macro with wikify, context object also holds reference to tiddler containing the macro ("inTiddler"). Thanks to SimonBaird.
** Support Firefox 1.5.0.1
** Internal
*** Make "JSLint" conform
*** "Only install once"
* v1.0.4 (2006-01-06)
** Support TiddlyWiki 2.0
* v1.0.3 (2005-12-22)
** Features:
*** Write output to a file supports multi-byte environments (Thanks to Bram Chen)
*** Provide API to access the forEachTiddler functionality directly through JavaScript (see getTiddlers and performMacro)
** Enhancements:
*** Improved error messages on InternetExplorer.
* v1.0.2 (2005-12-10)
** Features:
*** context object also holds reference to store (TiddlyWiki)
** Fixed Bugs:
*** ForEachTiddler 1.0.1 has broken support on win32 Opera 8.51 (Thanks to BrunoSabin for reporting)
* v1.0.1 (2005-12-08)
** Features:
*** Access tiddlers stored in separated TiddlyWikis through the "in" option. I.e. you are no longer limited to only work on the "current TiddlyWiki".
*** Write output to an external file using the "toFile" option of the "write" action. With this option you may write your customized tiddler exports.
*** Use the "script" section to define "helper" JavaScript functions etc. to be used in the various JavaScript expressions (whereClause, sortClause, action arguments,...).
*** Access and store context information for the current forEachTiddler invocation (through the build-in "context" object) .
*** Improved script evaluation (for where/sort clause and write scripts).
* v1.0.0 (2005-11-20)
** initial version
!Code
***/
//{{{
//============================================================================
//============================================================================
// ForEachTiddlerPlugin
//============================================================================
//============================================================================
// Only install once
if (!version.extensions.ForEachTiddlerPlugin) {
if (!window.abego) window.abego = {};
version.extensions.ForEachTiddlerPlugin = {
major: 1, minor: 0, revision: 8,
date: new Date(2007,3,12),
source: "http://tiddlywiki.abego-software.de/#ForEachTiddlerPlugin",
licence: "[[BSD open source license (abego Software)|http://www.abego-software.de/legal/apl-v10.html]]",
copyright: "Copyright (c) abego Software GmbH, 2005-2007 (www.abego-software.de)"
};
// For backward compatibility with TW 1.2.x
//
if (!TiddlyWiki.prototype.forEachTiddler) {
TiddlyWiki.prototype.forEachTiddler = function(callback) {
for(var t in this.tiddlers) {
callback.call(this,t,this.tiddlers[t]);
}
};
}
//============================================================================
// forEachTiddler Macro
//============================================================================
version.extensions.forEachTiddler = {
major: 1, minor: 0, revision: 8, date: new Date(2007,3,12), provider: "http://tiddlywiki.abego-software.de"};
// ---------------------------------------------------------------------------
// Configurations and constants
// ---------------------------------------------------------------------------
config.macros.forEachTiddler = {
// Standard Properties
label: "forEachTiddler",
prompt: "Perform actions on a (sorted) selection of tiddlers",
// actions
actions: {
addToList: {},
write: {}
}
};
// ---------------------------------------------------------------------------
// The forEachTiddler Macro Handler
// ---------------------------------------------------------------------------
config.macros.forEachTiddler.getContainingTiddler = function(e) {
while(e && !hasClass(e,"tiddler"))
e = e.parentNode;
var title = e ? e.getAttribute("tiddler") : null;
return title ? store.getTiddler(title) : null;
};
config.macros.forEachTiddler.handler = function(place,macroName,params,wikifier,paramString,tiddler) {
// config.macros.forEachTiddler.traceMacroCall(place,macroName,params,wikifier,paramString,tiddler);
if (!tiddler) tiddler = config.macros.forEachTiddler.getContainingTiddler(place);
// --- Parsing ------------------------------------------
var i = 0; // index running over the params
// Parse the "in" clause
var tiddlyWikiPath = undefined;
if ((i < params.length) && params[i] == "in") {
i++;
if (i >= params.length) {
this.handleError(place, "TiddlyWiki path expected behind 'in'.");
return;
}
tiddlyWikiPath = this.paramEncode((i < params.length) ? params[i] : "");
i++;
}
// Parse the where clause
var whereClause ="true";
if ((i < params.length) && params[i] == "where") {
i++;
whereClause = this.paramEncode((i < params.length) ? params[i] : "");
i++;
}
// Parse the sort stuff
var sortClause = null;
var sortAscending = true;
if ((i < params.length) && params[i] == "sortBy") {
i++;
if (i >= params.length) {
this.handleError(place, "sortClause missing behind 'sortBy'.");
return;
}
sortClause = this.paramEncode(params[i]);
i++;
if ((i < params.length) && (params[i] == "ascending" || params[i] == "descending")) {
sortAscending = params[i] == "ascending";
i++;
}
}
// Parse the script
var scriptText = null;
if ((i < params.length) && params[i] == "script") {
i++;
scriptText = this.paramEncode((i < params.length) ? params[i] : "");
i++;
}
// Parse the action.
// When we are already at the end use the default action
var actionName = "addToList";
if (i < params.length) {
if (!config.macros.forEachTiddler.actions[params[i]]) {
this.handleError(place, "Unknown action '"+params[i]+"'.");
return;
} else {
actionName = params[i];
i++;
}
}
// Get the action parameter
// (the parsing is done inside the individual action implementation.)
var actionParameter = params.slice(i);
// --- Processing ------------------------------------------
try {
this.performMacro({
place: place,
inTiddler: tiddler,
whereClause: whereClause,
sortClause: sortClause,
sortAscending: sortAscending,
actionName: actionName,
actionParameter: actionParameter,
scriptText: scriptText,
tiddlyWikiPath: tiddlyWikiPath});
} catch (e) {
this.handleError(place, e);
}
};
// Returns an object with properties "tiddlers" and "context".
// tiddlers holds the (sorted) tiddlers selected by the parameter,
// context the context of the execution of the macro.
//
// The action is not yet performed.
//
// @parameter see performMacro
//
config.macros.forEachTiddler.getTiddlersAndContext = function(parameter) {
var context = config.macros.forEachTiddler.createContext(parameter.place, parameter.whereClause, parameter.sortClause, parameter.sortAscending, parameter.actionName, parameter.actionParameter, parameter.scriptText, parameter.tiddlyWikiPath, parameter.inTiddler);
var tiddlyWiki = parameter.tiddlyWikiPath ? this.loadTiddlyWiki(parameter.tiddlyWikiPath) : store;
context["tiddlyWiki"] = tiddlyWiki;
// Get the tiddlers, as defined by the whereClause
var tiddlers = this.findTiddlers(parameter.whereClause, context, tiddlyWiki);
context["tiddlers"] = tiddlers;
// Sort the tiddlers, when sorting is required.
if (parameter.sortClause) {
this.sortTiddlers(tiddlers, parameter.sortClause, parameter.sortAscending, context);
}
return {tiddlers: tiddlers, context: context};
};
// Returns the (sorted) tiddlers selected by the parameter.
//
// The action is not yet performed.
//
// @parameter see performMacro
//
config.macros.forEachTiddler.getTiddlers = function(parameter) {
return this.getTiddlersAndContext(parameter).tiddlers;
};
// Performs the macros with the given parameter.
//
// @param parameter holds the parameter of the macro as separate properties.
// The following properties are supported:
//
// place
// whereClause
// sortClause
// sortAscending
// actionName
// actionParameter
// scriptText
// tiddlyWikiPath
//
// All properties are optional.
// For most actions the place property must be defined.
//
config.macros.forEachTiddler.performMacro = function(parameter) {
var tiddlersAndContext = this.getTiddlersAndContext(parameter);
// Perform the action
var actionName = parameter.actionName ? parameter.actionName : "addToList";
var action = config.macros.forEachTiddler.actions[actionName];
if (!action) {
this.handleError(parameter.place, "Unknown action '"+actionName+"'.");
return;
}
var actionHandler = action.handler;
actionHandler(parameter.place, tiddlersAndContext.tiddlers, parameter.actionParameter, tiddlersAndContext.context);
};
// ---------------------------------------------------------------------------
// The actions
// ---------------------------------------------------------------------------
// Internal.
//
// --- The addToList Action -----------------------------------------------
//
config.macros.forEachTiddler.actions.addToList.handler = function(place, tiddlers, parameter, context) {
// Parse the parameter
var p = 0;
// Check for extra parameters
if (parameter.length > p) {
config.macros.forEachTiddler.createExtraParameterErrorElement(place, "addToList", parameter, p);
return;
}
// Perform the action.
var list = document.createElement("ul");
place.appendChild(list);
for (var i = 0; i < tiddlers.length; i++) {
var tiddler = tiddlers[i];
var listItem = document.createElement("li");
list.appendChild(listItem);
createTiddlyLink(listItem, tiddler.title, true);
}
};
abego.parseNamedParameter = function(name, parameter, i) {
var beginExpression = null;
if ((i < parameter.length) && parameter[i] == name) {
i++;
if (i >= parameter.length) {
throw "Missing text behind '%0'".format([name]);
}
return config.macros.forEachTiddler.paramEncode(parameter[i]);
}
return null;
}
// Internal.
//
// --- The write Action ---------------------------------------------------
//
config.macros.forEachTiddler.actions.write.handler = function(place, tiddlers, parameter, context) {
// Parse the parameter
var p = 0;
if (p >= parameter.length) {
this.handleError(place, "Missing expression behind 'write'.");
return;
}
var textExpression = config.macros.forEachTiddler.paramEncode(parameter[p]);
p++;
// Parse the "begin" option
var beginExpression = abego.parseNamedParameter("begin", parameter, p);
if (beginExpression !== null)
p += 2;
var endExpression = abego.parseNamedParameter("end", parameter, p);
if (endExpression !== null)
p += 2;
var noneExpression = abego.parseNamedParameter("none", parameter, p);
if (noneExpression !== null)
p += 2;
// Parse the "toFile" option
var filename = null;
var lineSeparator = undefined;
if ((p < parameter.length) && parameter[p] == "toFile") {
p++;
if (p >= parameter.length) {
this.handleError(place, "Filename expected behind 'toFile' of 'write' action.");
return;
}
filename = config.macros.forEachTiddler.getLocalPath(config.macros.forEachTiddler.paramEncode(parameter[p]));
p++;
if ((p < parameter.length) && parameter[p] == "withLineSeparator") {
p++;
if (p >= parameter.length) {
this.handleError(place, "Line separator text expected behind 'withLineSeparator' of 'write' action.");
return;
}
lineSeparator = config.macros.forEachTiddler.paramEncode(parameter[p]);
p++;
}
}
// Check for extra parameters
if (parameter.length > p) {
config.macros.forEachTiddler.createExtraParameterErrorElement(place, "write", parameter, p);
return;
}
// Perform the action.
var func = config.macros.forEachTiddler.getEvalTiddlerFunction(textExpression, context);
var count = tiddlers.length;
var text = "";
if (count > 0 && beginExpression)
text += config.macros.forEachTiddler.getEvalTiddlerFunction(beginExpression, context)(undefined, context, count, undefined);
for (var i = 0; i < count; i++) {
var tiddler = tiddlers[i];
text += func(tiddler, context, count, i);
}
if (count > 0 && endExpression)
text += config.macros.forEachTiddler.getEvalTiddlerFunction(endExpression, context)(undefined, context, count, undefined);
if (count == 0 && noneExpression)
text += config.macros.forEachTiddler.getEvalTiddlerFunction(noneExpression, context)(undefined, context, count, undefined);
if (filename) {
if (lineSeparator !== undefined) {
lineSeparator = lineSeparator.replace(/\\n/mg, "\n").replace(/\\r/mg, "\r");
text = text.replace(/\n/mg,lineSeparator);
}
saveFile(filename, convertUnicodeToUTF8(text));
} else {
var wrapper = createTiddlyElement(place, "span");
wikify(text, wrapper, null/* highlightRegExp */, context.inTiddler);
}
};
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
// Internal.
//
config.macros.forEachTiddler.createContext = function(placeParam, whereClauseParam, sortClauseParam, sortAscendingParam, actionNameParam, actionParameterParam, scriptText, tiddlyWikiPathParam, inTiddlerParam) {
return {
place : placeParam,
whereClause : whereClauseParam,
sortClause : sortClauseParam,
sortAscending : sortAscendingParam,
script : scriptText,
actionName : actionNameParam,
actionParameter : actionParameterParam,
tiddlyWikiPath : tiddlyWikiPathParam,
inTiddler : inTiddlerParam, // the tiddler containing the <<forEachTiddler ...>> macro call.
viewerTiddler : config.macros.forEachTiddler.getContainingTiddler(placeParam) // the tiddler showing the forEachTiddler result
};
};
// Internal.
//
// Returns a TiddlyWiki with the tiddlers loaded from the TiddlyWiki of
// the given path.
//
config.macros.forEachTiddler.loadTiddlyWiki = function(path, idPrefix) {
if (!idPrefix) {
idPrefix = "store";
}
var lenPrefix = idPrefix.length;
// Read the content of the given file
var content = loadFile(this.getLocalPath(path));
if(content === null) {
throw "TiddlyWiki '"+path+"' not found.";
}
var tiddlyWiki = new TiddlyWiki();
// Starting with TW 2.2 there is a helper function to import the tiddlers
if (tiddlyWiki.importTiddlyWiki) {
if (!tiddlyWiki.importTiddlyWiki(content))
throw "File '"+path+"' is not a TiddlyWiki.";
tiddlyWiki.dirty = false;
return tiddlyWiki;
}
// The legacy code, for TW < 2.2
// Locate the storeArea div's
var posOpeningDiv = content.indexOf(startSaveArea);
var posClosingDiv = content.lastIndexOf(endSaveArea);
if((posOpeningDiv == -1) || (posClosingDiv == -1)) {
throw "File '"+path+"' is not a TiddlyWiki.";
}
var storageText = content.substr(posOpeningDiv + startSaveArea.length, posClosingDiv);
// Create a "div" element that contains the storage text
var myStorageDiv = document.createElement("div");
myStorageDiv.innerHTML = storageText;
myStorageDiv.normalize();
// Create all tiddlers in a new TiddlyWiki
// (following code is modified copy of TiddlyWiki.prototype.loadFromDiv)
var store = myStorageDiv.childNodes;
for(var t = 0; t < store.length; t++) {
var e = store[t];
var title = null;
if(e.getAttribute)
title = e.getAttribute("tiddler");
if(!title && e.id && e.id.substr(0,lenPrefix) == idPrefix)
title = e.id.substr(lenPrefix);
if(title && title !== "") {
var tiddler = tiddlyWiki.createTiddler(title);
tiddler.loadFromDiv(e,title);
}
}
tiddlyWiki.dirty = false;
return tiddlyWiki;
};
// Internal.
//
// Returns a function that has a function body returning the given javaScriptExpression.
// The function has the parameters:
//
// (tiddler, context, count, index)
//
config.macros.forEachTiddler.getEvalTiddlerFunction = function (javaScriptExpression, context) {
var script = context["script"];
var functionText = "var theFunction = function(tiddler, context, count, index) { return "+javaScriptExpression+"}";
var fullText = (script ? script+";" : "")+functionText+";theFunction;";
return eval(fullText);
};
// Internal.
//
config.macros.forEachTiddler.findTiddlers = function(whereClause, context, tiddlyWiki) {
var result = [];
var func = config.macros.forEachTiddler.getEvalTiddlerFunction(whereClause, context);
tiddlyWiki.forEachTiddler(function(title,tiddler) {
if (func(tiddler, context, undefined, undefined)) {
result.push(tiddler);
}
});
return result;
};
// Internal.
//
config.macros.forEachTiddler.createExtraParameterErrorElement = function(place, actionName, parameter, firstUnusedIndex) {
var message = "Extra parameter behind '"+actionName+"':";
for (var i = firstUnusedIndex; i < parameter.length; i++) {
message += " "+parameter[i];
}
this.handleError(place, message);
};
// Internal.
//
config.macros.forEachTiddler.sortAscending = function(tiddlerA, tiddlerB) {
var result =
(tiddlerA.forEachTiddlerSortValue == tiddlerB.forEachTiddlerSortValue)
? 0
: (tiddlerA.forEachTiddlerSortValue < tiddlerB.forEachTiddlerSortValue)
? -1
: +1;
return result;
};
// Internal.
//
config.macros.forEachTiddler.sortDescending = function(tiddlerA, tiddlerB) {
var result =
(tiddlerA.forEachTiddlerSortValue == tiddlerB.forEachTiddlerSortValue)
? 0
: (tiddlerA.forEachTiddlerSortValue < tiddlerB.forEachTiddlerSortValue)
? +1
: -1;
return result;
};
// Internal.
//
config.macros.forEachTiddler.sortTiddlers = function(tiddlers, sortClause, ascending, context) {
// To avoid evaluating the sortClause whenever two items are compared
// we pre-calculate the sortValue for every item in the array and store it in a
// temporary property ("forEachTiddlerSortValue") of the tiddlers.
var func = config.macros.forEachTiddler.getEvalTiddlerFunction(sortClause, context);
var count = tiddlers.length;
var i;
for (i = 0; i < count; i++) {
var tiddler = tiddlers[i];
tiddler.forEachTiddlerSortValue = func(tiddler,context, undefined, undefined);
}
// Do the sorting
tiddlers.sort(ascending ? this.sortAscending : this.sortDescending);
// Delete the temporary property that holds the sortValue.
for (i = 0; i < tiddlers.length; i++) {
delete tiddlers[i].forEachTiddlerSortValue;
}
};
// Internal.
//
config.macros.forEachTiddler.trace = function(message) {
displayMessage(message);
};
// Internal.
//
config.macros.forEachTiddler.traceMacroCall = function(place,macroName,params) {
var message ="<<"+macroName;
for (var i = 0; i < params.length; i++) {
message += " "+params[i];
}
message += ">>";
displayMessage(message);
};
// Internal.
//
// Creates an element that holds an error message
//
config.macros.forEachTiddler.createErrorElement = function(place, exception) {
var message = (exception.description) ? exception.description : exception.toString();
return createTiddlyElement(place,"span",null,"forEachTiddlerError","<<forEachTiddler ...>>: "+message);
};
// Internal.
//
// @param place [may be null]
//
config.macros.forEachTiddler.handleError = function(place, exception) {
if (place) {
this.createErrorElement(place, exception);
} else {
throw exception;
}
};
// Internal.
//
// Encodes the given string.
//
// Replaces
// "$))" to ">>"
// "$)" to ">"
//
config.macros.forEachTiddler.paramEncode = function(s) {
var reGTGT = new RegExp("\\$\\)\\)","mg");
var reGT = new RegExp("\\$\\)","mg");
return s.replace(reGTGT, ">>").replace(reGT, ">");
};
// Internal.
//
// Returns the given original path (that is a file path, starting with "file:")
// as a path to a local file, in the systems native file format.
//
// Location information in the originalPath (i.e. the "#" and stuff following)
// is stripped.
//
config.macros.forEachTiddler.getLocalPath = function(originalPath) {
// Remove any location part of the URL
var hashPos = originalPath.indexOf("#");
if(hashPos != -1)
originalPath = originalPath.substr(0,hashPos);
// Convert to a native file format assuming
// "file:///x:/path/path/path..." - pc local file --> "x:\path\path\path..."
// "file://///server/share/path/path/path..." - FireFox pc network file --> "\\server\share\path\path\path..."
// "file:///path/path/path..." - mac/unix local file --> "/path/path/path..."
// "file://server/share/path/path/path..." - pc network file --> "\\server\share\path\path\path..."
var localPath;
if(originalPath.charAt(9) == ":") // pc local file
localPath = unescape(originalPath.substr(8)).replace(new RegExp("/","g"),"\\");
else if(originalPath.indexOf("file://///") === 0) // FireFox pc network file
localPath = "\\\\" + unescape(originalPath.substr(10)).replace(new RegExp("/","g"),"\\");
else if(originalPath.indexOf("file:///") === 0) // mac/unix local file
localPath = unescape(originalPath.substr(7));
else if(originalPath.indexOf("file:/") === 0) // mac/unix local file
localPath = unescape(originalPath.substr(5));
else // pc network file
localPath = "\\\\" + unescape(originalPath.substr(7)).replace(new RegExp("/","g"),"\\");
return localPath;
};
// ---------------------------------------------------------------------------
// Stylesheet Extensions (may be overridden by local StyleSheet)
// ---------------------------------------------------------------------------
//
setStylesheet(
".forEachTiddlerError{color: #ffffff;background-color: #880000;}",
"forEachTiddler");
//============================================================================
// End of forEachTiddler Macro
//============================================================================
//============================================================================
// String.startsWith Function
//============================================================================
//
// Returns true if the string starts with the given prefix, false otherwise.
//
version.extensions["String.startsWith"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
String.prototype.startsWith = function(prefix) {
var n = prefix.length;
return (this.length >= n) && (this.slice(0, n) == prefix);
};
//============================================================================
// String.endsWith Function
//============================================================================
//
// Returns true if the string ends with the given suffix, false otherwise.
//
version.extensions["String.endsWith"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
String.prototype.endsWith = function(suffix) {
var n = suffix.length;
return (this.length >= n) && (this.right(n) == suffix);
};
//============================================================================
// String.contains Function
//============================================================================
//
// Returns true when the string contains the given substring, false otherwise.
//
version.extensions["String.contains"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
String.prototype.contains = function(substring) {
return this.indexOf(substring) >= 0;
};
//============================================================================
// Array.indexOf Function
//============================================================================
//
// Returns the index of the first occurance of the given item in the array or
// -1 when no such item exists.
//
// @param item [may be null]
//
version.extensions["Array.indexOf"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
Array.prototype.indexOf = function(item) {
for (var i = 0; i < this.length; i++) {
if (this[i] == item) {
return i;
}
}
return -1;
};
//============================================================================
// Array.contains Function
//============================================================================
//
// Returns true when the array contains the given item, otherwise false.
//
// @param item [may be null]
//
version.extensions["Array.contains"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
Array.prototype.contains = function(item) {
return (this.indexOf(item) >= 0);
};
//============================================================================
// Array.containsAny Function
//============================================================================
//
// Returns true when the array contains at least one of the elements
// of the item. Otherwise (or when items contains no elements) false is returned.
//
version.extensions["Array.containsAny"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
Array.prototype.containsAny = function(items) {
for(var i = 0; i < items.length; i++) {
if (this.contains(items[i])) {
return true;
}
}
return false;
};
//============================================================================
// Array.containsAll Function
//============================================================================
//
// Returns true when the array contains all the items, otherwise false.
//
// When items is null false is returned (even if the array contains a null).
//
// @param items [may be null]
//
version.extensions["Array.containsAll"] = {major: 1, minor: 0, revision: 0, date: new Date(2005,11,20), provider: "http://tiddlywiki.abego-software.de"};
//
Array.prototype.containsAll = function(items) {
for(var i = 0; i < items.length; i++) {
if (!this.contains(items[i])) {
return false;
}
}
return true;
};
} // of "install only once"
// Used Globals (for JSLint) ==============
// ... DOM
/*global document */
// ... TiddlyWiki Core
/*global convertUnicodeToUTF8, createTiddlyElement, createTiddlyLink,
displayMessage, endSaveArea, hasClass, loadFile, saveFile,
startSaveArea, store, wikify */
//}}}
/***
!Licence and Copyright
Copyright (c) abego Software ~GmbH, 2005 ([[www.abego-software.de|http://www.abego-software.de]])
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or other
materials provided with the distribution.
Neither the name of abego Software nor the names of its contributors may be
used to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
***/
Ganglia is a distributed monitoring system orientated to high performance computing clusters (HPC). If you run an HPC system you need Ganglia. Ganglia lets you navigate amongst heaps of overview graphs, dashboards and of course detail graphs.
| [img[Ganglia Graph|images/ganglia.png]] |
| Example of a Ganglia Overview Graph |
|borderless|k
You can get Ganglia from http://ganglia.sourceforge.net/.
The Hobbit is a fork of [[BigBrother]]. It is BigBrother reinvented and reinvigorated. The Hobbit Monitor was developed by the same person who was responsible for the ~BBGen BigBrother extension and it is compatible with the majority of BigBrother extensions and plugins. The Hobbit is now called [[Xymon]].
You can get The Hobbit at http://hobbitmon.sourceforge.net/.
This is a collection of my mini guides and tricks and traps. Click on one of the links on the left to open a topic.
/***
|Name|InlineJavascriptPlugin|
|Source|http://www.TiddlyTools.com/#InlineJavascriptPlugin|
|Documentation|http://www.TiddlyTools.com/#InlineJavascriptPluginInfo|
|Version|1.9.2|
|Author|Eric Shulman - ELS Design Studios|
|License|http://www.TiddlyTools.com/#LegalStatements <br>and [[Creative Commons Attribution-ShareAlike 2.5 License|http://creativecommons.org/licenses/by-sa/2.5/]]|
|~CoreVersion|2.1|
|Type|plugin|
|Requires||
|Overrides||
|Description|Insert Javascript executable code directly into your tiddler content.|
''Call directly into TW core utility routines, define new functions, calculate values, add dynamically-generated TiddlyWiki-formatted output'' into tiddler content, or perform any other programmatic actions each time the tiddler is rendered.
!!!!!Documentation
>see [[InlineJavascriptPluginInfo]]
!!!!!Revisions
<<<
2008.03.03 [1.9.2] corrected declaration of wikifyPlainText() for 'TW 2.1.x compatibility fallback' (fixes Safari "parse error")
2008.02.23 [1.9.1] in onclick function, use string instead of array for 'bufferedHTML' attribute on link element (fixes IE errors)
2008.02.21 [1.9.0] 'onclick' scripts now allow returned text (or document.write() calls) to be wikified into a span that immediately follows the onclick link. Also, added default 'return false' handling if no return value provided (prevents HREF from being triggered -- return TRUE to allow HREF to be processed). Thanks to Xavier Verges for suggestion and preliminary code.
|please see [[InlineJavascriptPluginInfo]] for additional revision details|
2005.11.08 [1.0.0] initial release
<<<
!!!!!Code
***/
//{{{
version.extensions.inlineJavascript= {major: 1, minor: 9, revision: 2, date: new Date(2008,3,3)};
config.formatters.push( {
name: "inlineJavascript",
match: "\\<script",
lookahead: "\\<script(?: src=\\\"((?:.|\\n)*?)\\\")?(?: label=\\\"((?:.|\\n)*?)\\\")?(?: title=\\\"((?:.|\\n)*?)\\\")?(?: key=\\\"((?:.|\\n)*?)\\\")?( show)?\\>((?:.|\\n)*?)\\</script\\>",
handler: function(w) {
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
var src=lookaheadMatch[1];
var label=lookaheadMatch[2];
var tip=lookaheadMatch[3];
var key=lookaheadMatch[4];
var show=lookaheadMatch[5];
var code=lookaheadMatch[6];
if (src) { // load a script library
// make script tag, set src, add to body to execute, then remove for cleanup
var script = document.createElement("script"); script.src = src;
document.body.appendChild(script); document.body.removeChild(script);
}
if (code) { // there is script code
if (show) // show inline script code in tiddler output
wikify("{{{\n"+lookaheadMatch[0]+"\n}}}\n",w.output);
if (label) { // create a link to an 'onclick' script
// add a link, define click handler, save code in link (pass 'place'), set link attributes
var link=createTiddlyElement(w.output,"a",null,"tiddlyLinkExisting",wikifyPlainText(label));
var fixup=code.replace(/document.write\s*\(/gi,'place.bufferedHTML+=(');
link.code="function _out(place){"+fixup+"\n};_out(this);"
link.tiddler=w.tiddler;
link.onclick=function(){
this.bufferedHTML="";
try{ var r=eval(this.code);
if(this.bufferedHTML.length || (typeof(r)==="string")&&r.length)
var s=this.parentNode.insertBefore(document.createElement("span"),this.nextSibling);
if(this.bufferedHTML.length)
s.innerHTML=this.bufferedHTML;
if((typeof(r)==="string")&&r.length) {
wikify(r,s,null,this.tiddler);
return false;
} else return r!==undefined?r:false;
} catch(e){alert(e.description||e.toString());return false;}
};
link.setAttribute("title",tip||"");
var URIcode='javascript:void(eval(decodeURIComponent(%22(function(){try{';
URIcode+=encodeURIComponent(encodeURIComponent(code.replace(/\n/g,' ')));
URIcode+='}catch(e){alert(e.description||e.toString())}})()%22)))';
link.setAttribute("href",URIcode);
link.style.cursor="pointer";
if (key) link.accessKey=key.substr(0,1); // single character only
}
else { // run inline script code
var fixup=code.replace(/document.write\s*\(/gi,'place.innerHTML+=(');
var code="function _out(place){"+fixup+"\n};_out(w.output);"
try { var out=eval(code); } catch(e) { out=e.description?e.description:e.toString(); }
if (out && out.length) wikify(out,w.output,w.highlightRegExp,w.tiddler);
}
}
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
}
} )
//}}}
// // Backward-compatibility for TW2.1.x and earlier
//{{{
if (typeof(wikifyPlainText)=="undefined") window.wikifyPlainText=function(text,limit,tiddler) {
if(limit > 0) text = text.substr(0,limit);
var wikifier = new Wikifier(text,formatter,null,tiddler);
return wikifier.wikifyPlain();
}
//}}}
MRTG stands for 'Multi Router Traffic Grapher'. It is the grand-daddy of all opensource performance management tools. MRTG had serious performance issues when recording many data sources however porting to RRDTool instead of its native data format has given MRTG a new lease of life. MRTG has a plethora of addons that greatly expand its usefulness.
| [img[Sample MRTG Page|images/mrtg.jpg]] |
| Sample MRTG Page |
|borderless|k
MRTG has the restriction that it must graph two and only two datasources.
You can get MRTG from http://oss.oetiker.ch/mrtg/
Produces message statistics of postfix and sendmail mail servers by processing the log files.
| [img[Sample Mailgraph Graph|images/mailgraph.png]] |
| Sample Mailgraph Graph |
|borderless|k
You can get Mailgraph from http://mailgraph.schweikert.ch/
<html><a
href="javascript:void(0)"
onclick="story.closeAllTiddlers();story.displayTiddlers(null,store.getTiddlerText('DefaultTiddlers').readBracketedList())"
>Home</a></html>
[[What's New?]]
[[About]]
[[My Projects]]
[[My Papers]]
[[Network Management Model]]
[[Tools]]
[[Resources]]
[[How To]]
[[Event Handling]]
[[RSS]]
^^[[TiddlyWiki|http://www.tiddlywiki.com/]] <<version>>
There are two large scale active performance monitoring projects that I am aware of ...
*[[IEPM-BW|http://www-iepm.slac.stanford.edu/bw/]]
*[[perfSONAR|http://www.perfsonar.net/]]
Both use distributed active network probes to measure network bandwidth, find network bottlenecks and record routes. They are effectively trying to measure the performance of the Internet.
![[IEPM-BW|http://www-iepm.slac.stanford.edu/bw/]]
~IEPM-BW is a [[SLAC|http://www.slac.stanford.edu/]] project. IEPM stands for Internet End to end Performance Monitoring. The stated purpose of ~IEPM-BW is to support the Internet High Energy / Nuclear Physics communitee.
~IEPM-BW conducts active performance measurement from remote hosts in the Internet to a central collection point at SLAC. The measurement software on the remote hosts is controlled from SLAC using SSH. Accounts on the remote hosts are made available to SLAC for this purpose by participating organisations.
Measurements are made using software such as ~PingER and [[iperf]] for traffic and latency measurements bbcp and pipechar for bulk transfer statistics and traceroute for discovering topology. This information is presented on a series of scatter plots and graphs and some that attempt to combine many disparate types of information.
![[perfSONAR|http://www.perfsonar.net/]]
perfSONAR is a framework for the collection and publishing of network performance data. It is currently targeting the requirements of the Large Hadron Collider communitee. The architecture of perfSONAR supports collaboration by the communitee it serves in development and the construction and operation of the perfSONAR infrastructure. A key element is that services in the framework do not have to be homogeneous or operated by a single organisation.
perfSONAR is made up of a series of web services...
* the Measurement Point Service provides measurement data
* the Measurement Archive Service publishes historical monitoring data
* the Information Service enables users to find services and provides a description of those services, it also provides network topology information
* the Authentication Service can be used to restrict access to service capabilities
* the Transformation Service transfers and modifies data between services
* the Resource Protector Service arbitrates the consumption of resources
Performance measurements are made between measurement points on a scheduled or controlled ad-hoc basis. The measurement tools used include ~PingER OWAMP and [[iperf]].
The heart of Mon is a general purpose scheduler. It also contains alert management and network testing functionality. Mon is one of the older freeware network management tools. Releases date back to 1997. However Mon is still under active development.
Mon is a bit like a meccano kit for constructing a fault management system around the central scheduler. The author has seen Mon used as a redundant server failover controller for example. All the network tests and the alerting are modular extensions to the scheduler. Mon has explicit support for service and system dependencies. This is an integral part of the configuration language.
Mon is fundamentally a text driven tool that alerts administrators to fault conditions using alphanumeric pager protocol. However several people have produced web based front ends for Mon. Mon is written in Perl and as such should run on most platforms.
You can find more information on Mon [[here|http://www.kernel.org/software/mon/]].
Monit is a small simple, but useful automaton that can monitor correct operation of a UNIX system and has some capability of monitoring remote systems. Monit can check for correct operation of processes and restart them if necessary for example. All checks and actions are configurable in an easy scripting language. Monit has a built in web server. You can get more information on Monit [[here|http://mmonit.com/monit/]]
Munin is a server performance management tool. Munin nodes run on monitored servers and report back across the network to the Munin server. Munin has plugins to cover just about every aspect of server performance. My colleague Stephen uses and thoroughly endorses Munin. You can read more about it [[here|http://munin.projects.linpro.no/]].
{{center{
[img[Example Munin Graph|images/munin-iostat-day.png]]
Example Munin Graph
}}}
Recently I have been working on a system to combine SNMP traps and Syslog messages and link them to my fault management system. The system I am working on uses the snmptrapd from the ~NetSNMP project, the SNMP Trap Translator SNMPTT, SEC the simple event correlator and a system I am writing called bb-event.
Snmptrapd is used to receive SNMP traps and queue them for processing by SNMPTT. SNMPTT is used to format the traps in plain language. The output of SNMPTT is sent to syslog. The advantage of using SNMPTT is that it produces a plain language text log that I can use as an audit trail. While I was working on trap handling I found a problem with Cisco traps [[Cisco SNMP traps and SNMPTT]].
Syslog is used as a central collector to log messages from SNMPTT and syslog messages sent by servers and network equipment. Syslog saves the messages in log files and sends them to SEC for further processing. I am currently using the standard BSD syslog, but plan to change to syslog-ng sometime soon so that I can use a named pipe to send messages to SEC rather than tailing the text log file.
SEC is used to process event messages. It allows events to be filtered and grouped into composite events. I am using SEC currently to send alarms to my fault management system (~BigBrother using a system I am writing called bb-event) and emails. An artificial ignorance approach is being used to develop the correlation rules, I am sent an email for every message that SEC does not understand. There are also some events that I plan to handle solely by email such as system configurations.
bb-event turns momentary events into alarms with duration. It also allows multiple different events to be grouped in the same alarm.
//A Complete Network Management System using Freeware Tools// presented at the [[AUUG|http://www.auug.org.au/]] 2005 summer conference.
{{indent{[[paper|papers/auug2005/Freeware-NMS.pdf]] [[presentation|papers/auug2005/AUUG2005-CompleteNMS/AUUG2005-CompleteNMS.html]]
Here are some of the network management projects I have been working on recently.
You can find my project releases [[here|projects/]].
Nagios was originally known as ~NetSaint. It is probably the most popular opensource fault management system. Nagios has multiple screens to display different views of network status and to control Nagios.
| [img[Sample Nagios Page|images/nagios.jpg]] |
| Sample Nagios Page (from nagios.org) |
|borderless|k
Nagios integrates with [[Cacti]] for trending of data and several different alerting packages.
You can get Nagios from http://www.nagios.org.
A discotheque for your network.
Netdisco is a network discovery tool. Given a starting point it will discover all the switches and routers in your network and the connections between them. The main task that Netdisco performs is gathering all the MAC and IP addresses in your network and the port(s) that they are or have been connected to. It also provides an inventory of network devices and their software versions. A postgres database is used to store the iformation gathered by Netdisco. This is an essential tool if you use dynamic DHCP.
Netdisco works with network equipment from
*Allied Telesyn
*Asante
*Bay
*Cisco
*Foundry
*HP
*Nortel
*Proxim
*Zyxel
You can get it from http://www.netdisco.org
|It is helpful to think of network management as comprising the following five areas...|
|* [[Fault Management]]|
|* [[Performance Management]]|
|* [[Configuration Management]]|
|* Accounting Management|
|* [[Security Management]]|
|borderlessL|k
This is the ISO network management model. A complete Network Management System addresses each of these areas. You will find many of the opensource network management tools I know about listed in the area of this model that is most relevant to the particular tool.
Orca can graph every server stat that you have every thought about. A drawback of Orca is that the stats are stored on the filesystem of the machine that is being monitored so that you need to use an NFS or something like rsync if you don't want to run a web server on every machine Orca is monitoring.
You can get Orca from http://www.orcaware.com/orca/.
Type the text for 'New Tiddler'
Collecting network performance information in order to avoid network performance issues and to enable capacity planning.
!Performance Management Systems
These tools are general purpose performance management systems that will provide most of the functionally you need in a performance management system.
|*[[MRTG]] the earliest opensource performance management system|
|*[[Cricket]] a general purpose data collection and graphing tool|
|*[[Cacti]] claims to be the //universal graphing tool//, it is currently the most popular opensource performance management system|
|borderlessL|k
!The Underpinnings
*[[RRDTool]] is used by almost every opensource performance management tool
*[[Net-SNMP]] SNMP daemon, SNMP utilities and C & Perl API
!Specific Purpose Tools
!!Server Monitoring
*[[Orca]] server performance management
*[[Munin]] server performance management
*[[Ganglia]] cluster performance management
!!Application Monitoring
*[[Mailgraph]], [[Couriergraph]], [[Queuegraph]], [[amavis-stats]] mail stats
*[[Dnsgraph]] [[BIND 9.5 DNS Stats]] bind stats
!!Latency
*[[Smokeping]] the guerrilla performance management tool
!!Active Performance Measurement
*[[iperf]] and [[Thrulay]] are two tools for measuring network available bandwidth and jitter
!Graphing Addons
These tools can be used to suppliment one of the general purpose systems or to help you to build custom systems of your own.
*[[drraw]] CGI form generated templates and dashboards
*[[RRGrapher]] CGI dialogue generated graphs
[>img[petery|images/petery.jpg]]
I'm currently employed as a Senior Network Administrator in the [[University of Technology, Sydney|http://www.uts.edu.au/]] which is situated in Sydney, Australia. I use mainly opensource tools for network management at work and spend a lot of my time writing new tests and tools as well as integrating other peoples tools into my network management system.
My email address is //Peter d0t Yardley at uts d0t edu d0t au//.
[[QuickPage|http://www.qpage.org]] is an SNPP (Simple Network Pager Protocol) based system that will connect to a pager service using a dial-up modem with TAP/IXO. It has a client that sends pages to the server using SNPP, this means that the client does not have to be on the same machine as the server. The server then connects the to pager service using the dial-up modem. Many SMS service providers offer a TAP/IXO gateway to their SMS service.
One issue with ~QuickPage is that the client does not store and forward, so if the server is down paging fails, but in practice this system is quite reliable.
RANCID stands for 'Really Awesome New ~ConfIguration Differ', which is a bit of a stretch.
RANCID collects configuration information by periodically logging into network equipment and issuing CLI commands. Hardware configuration, software versions and device configuration are recorded. The configuration information is stored in a CVS repository to maintain a history of changes. Change reports are emailed at the end of each run where changes to network equipment configuration have been collected. RANCID comes with a [[Looking Glass]] that can be used to issue CLI commands to network equipment.
If you use a tool like RANCID that is allowed to login to network equipment it is a good idea to give it its own account and use [[TACACS+]] (if you are using compatible equipment) to restrict the commands that the account can issue.
You can get RANCID from http://www.shrubbery.net/rancid/
Round Robin Databases (RRD) are used to store performance data in most of the opensource performance management tools. The ~RRDs are efficient way or storing performance data in a fixed size file. ~RRDTool manages the storing of data in ~RRDs and producing graphs from that data.
An RRD consists of one or more data sources (DS). Each DS represents a data sequence and contains one or more Round Robin Archives. Each RRA stores a specified number of samples of the the data for a particular time period. When data is older than the RRA can store it is deleted. Data for longer time period ~RRAs comes by combining data points from ~RRAs with a shorter time period using a Consolidation Function (CF). The CF is usually something like average or maximum. A DS can have multiple ~RRAs for the longer time periods and they can use different Consolidation Functions so that an average and a maximum can be stored. A typical RRD will have ~RRAs for one day, one week, one month and one year periods. A problem with ~RRDs is that longer time period data loses detail, the impact of this can be reduced by storing maximums and/or minimums as well as averages.
| [img[RRD Consolidation|images/RRD.png]] |
| Data Consolidation in an RRD |
|borderless|k
~RRDTool has a set of command line tools as well as ~APIs for C, Perl, Python, Ruby, TCL and PHP.
You can get ~RRDTool from http://oss.oetiker.ch/rrdtool/.
RRGrapher lets you design graphs for RRD databases collected by other tools using a CGI dialogue. The graphs can then be summonsed up again by using the URL of the finished graph.
You can get RRGrapher from http://net.doit.wisc.edu/~plonka/RRGrapher/
You can get the RSS feed for this wiki [[here|index.xml]].
[[Peter Welcher's web site|http://www.netcraftsmen.net/welcher/]] check out his [[articles|http://www.netcraftsmen.net/welcher/papers/index.htm]] they really are good!
[[CAIDA|http://www.caida.org/home/about/]] has a [[taxonomy|http://www.caida.org/tools/]] of their network management tools.
[[SLAC|http://www.slac.stanford.edu/]] maintains a [[taxonomy|http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html]] of available network management tools organised by various useful classifications.
[[Henning Schulzrinne|http://www.cs.columbia.edu/~hgs/]] maintains a [[catalogue|http://www.cs.columbia.edu/~hgs/internet/tools.html]] of Internet measurement tools.
[[Open Source Testing|http://www.opensourcetesting.org/]] is a web site dedicated to "open source testing tools, news and discussion".
SEC stands for Simple Event Correlator
SEC generates actions based on events presented as single lines of text from files, pipes or named pipes. It can filter and combine events based on rules. The events can come from sources such as syslog, SNMP traps or any other text stream. The actions that can be taken include execution of a user specified shell script or program, generation of new events and writing text output.
SEC can be found at http://kodu.neti.ee/~risto/sec/
I have written an event handler for [[SMS Server Tools]] that does acknowledgement of [[BigBrother]] alerts using SMS. You can find it on my [[projects page|projects/]].
SMS Server Tools is a system to send and receive SMS messages from a computer using a GSM modem or modems. It has a command line program to send SMS messages. Received messages are stored in a spool directory. There is provision to add an event handler to do extra processing and an alarm handler to process errors.
The [[first two versions|http://www.meinemullemaus.de/smstools/index-en.html]] were by Stefan Frings. This code was subsequently further developed to produce [[SMS Server Tools 3|http://smstools3.kekekasvi.com/]]
[[SMSLink|http://smslink.sourceforge.net/]] is a system to send and receive SMS messages from a computer using a GSM modem. This system is however network based. The client and server use the network for communication so they don't have to run on the same computer.
The server only runs on UNIX but the client can run on Windows.
SNMPTT stands for SNMP Trap Translator.
SNMP traps are not meant to be read by human eyes. SNMP traps are meant to be read by machines. SNMPTT is a machine for reading SNMP traps.
SNMPTT is passed SNMP traps from a trap handler such as snmptrapd from ~NetSNMP and translates them into plain language according to user specified rules. SNMPTT can also execute programs in response to traps received.
SNMPTT can be found at http://www.snmptt.org/
Controlling access to network resources and preventing the damage or disruption of those resources.
/***
|Name|ShowUpdatesPlugin|
|Created by|SaqImtiaz|
|Version|0.2 |
|Requires|~TW2.x|
!!!Description:
Allows you to list tiddlers that have changed since the users last visit. You can list only all changed tiddlers, or filter them to only show tiddlers that have or do not have a specific tag. By default a simple list of the titles of changed tiddlers is created. However, using an extremely versatile syntax you can provide a custom template for the generated text.
!!!Examples:
[[ShowUpdatesDocs]]
!!!Installation:
Copy the contents of this tiddler to your TW, tag with systemConfig, save and reload your TW.
!!!Syntax:
{{{<<showUpdates>>}}}
additional optional params:
{{{<showUpdates excludeTag:TagToExclude onlyTag:TagToList maxEntries:10 write:CustomWriteParameter >>}}}
excludeTag: ~TagToExclude
onlyTag: ~TagToList
maxEntries: max number of entries displayed when there are no updates. (default is 10, which can be changed in the config.macros.showUpdates.settings part of the code)
write: if a write parameter is not provided, an un-numbered list of the updates is generated. Alternatively, you can specify a custom 'template' for the text generated. The syntax for the write parameter is identical to that of the forEachTiddler macro. Additonal documentation on this syntax will be provided soon.
Some of the variables available in the write parameter are 'index', 'count' and 'lastVisit' where lastVisit is the date of the last visit in the format YYYYMMDDHHMM. Also areUpdates is a boolean that is true if there are new updates since the users last visit.
!!!To Do:
*refactor code to facilitate translations
*a streamlined version without the custom write parameter
!!!Code
***/
//{{{
window.lewcidLastVisit = '';
window.old_lewcid_whatsnew_restart = window.restart;
window.restart = function()
{
if(config.options.txtLastVisit)
lewcidLastVisit= config.options.txtLastVisit;
config.options.txtLastVisit = (new Date()).convertToYYYYMMDDHHMM();
saveOptionCookie('txtLastVisit');
window.old_lewcid_whatsnew_restart();
}
TiddlyWiki.prototype.lewcidGetTiddlers = function(field,excludeTag,includeTag,updatesOnly)
{
var results = [];
this.forEachTiddler(function(title,tiddler)
{
if(excludeTag == undefined || !tiddler.isTagged(excludeTag))
if(includeTag == undefined || tiddler.isTagged(includeTag))
if ( updatesOnly == false || tiddler.modified.convertToYYYYMMDDHHMM()>lewcidLastVisit)
results.push(tiddler);
});
if(field)
results.sort(function (a,b) {if(a[field] == b[field]) return(0); else return (a[field] < b[field]) ? -1 : +1; });
return results;
}
config.macros.showUpdates={};
config.macros.showUpdates.settings =
{
maxEntries: 10 //max items to show, if there are no updates since last visit
}
config.macros.showUpdates.handler = function(place,macroName,params,wikifier,paramString,tiddler)
{
var args = paramString.parseParams("list",null,true);
var write = getParam(args, "write", undefined);
var onlyTag = getParam(args, "onlyTag", undefined);
var excludeTag = getParam(args, "excludeTag", undefined);
var sortBy = "modified";
var maxEntries = getParam(args,"maxEntries",this.settings.maxEntries);
if (lewcidLastVisit)
{var tiddlers = store.lewcidGetTiddlers(sortBy,excludeTag,onlyTag,true);
var areUpdates = tiddlers.length>0? true:false;}
if (!lewcidLastVisit)
{var countLine = "!!Recent Updates:";
var tiddlers = store.lewcidGetTiddlers(sortBy,excludeTag,onlyTag,false);
var areUpdates = false;}
else if (tiddlers.length == 0)
{var countLine = "!!@@color:red;No new updates@@ since your last visit. @@color:#999;font-size:70%;" + (Date.convertFromYYYYMMDDHHMM(lewcidLastVisit)).formatString(" (DD/MM/YY)") + "@@\n!!Recent Updates:";
var tiddlers = store.lewcidGetTiddlers(sortBy,excludeTag,onlyTag,false);}
else
{var countLine ="!!@@color:red;"+ tiddlers.length + "@@ new " + (tiddlers.length==1?"update":"updates") + " since your last visit: @@color:#999;font-size:70%;" + (Date.convertFromYYYYMMDDHHMM(lewcidLastVisit)).formatString(" (DD/MM/YY)") + "@@";}
tiddlers = tiddlers.reverse();
var lastVisit = lewcidLastVisit? lewcidLastVisit:undefined;
var count = areUpdates == true? tiddlers.length : maxEntries;
var sp = createTiddlyElement(place,"span","showUpdates");
if (write==undefined)
{
wikify(countLine,sp);
var list = createTiddlyElement(sp,"ul");
for (var i = 0; i < count; i++)
{
var tiddler = tiddlers[i];
createTiddlyLink(createTiddlyElement(list,"li"), tiddler.title, true);
}
}
else
{
var list = '';
for (var index = 0; index < count; index++) {
var tiddler = tiddlers[index];
list += eval(write); }
wikify(list, sp);
}
}
//}}}
Opensource Network Management
http://members.iinet.com.au/~pyardley@ihug.com.au/
Smokeping measures the latency and packet loss of bursts of probes. Smokeping can test protocols, connectivity and applications. It fires off a burst of probes and graphs the result as a 'smoke cloud'. Smokeping can be used as a guerrilla performance application testing other peoples services. Want to know how you good your ISP really is, use Smokeping.
| [img[Sample Smokeping Graph|images/smokeping.png]] |
| Sample Smokeping Graph |
|borderless|k
You can get smokeping from http://oss.oetiker.ch/smokeping/.
.center {text-align:center;}
.borderless, .borderless table, .borderless td, .borderless tr, .borderless th, .borderless tbody { border:0 !important; margin:0 !important; padding:2px !important; td.vertical-align:top !important;margin-left: auto !important; margin-right: auto !important;}
.borderlessL, .borderlessL table, .borderlessL td, .borderlessL tr, .borderlessL th, .borderlessL tbody { border:0 !important; margin:0 !important; padding:2px !important; td.vertical-align:top !important;margin-left: 0 !important; margin-right: auto !important;}
.blog h2, .blog h3, .blog h4{
margin:0;
padding:0;
border-bottom:none;
}
.blog {margin-left:1.5em;}
.blog .excerpt {
margin:0;
margin-top:0.3em;
padding: 0;
margin-left:1em;
padding-left:1em;
font-size:90%;
border-left:1px solid #ddd;
}
~Syslog-NG is a powerful replacement for the standard BSD syslog server. The ~Syslog-NG configuration defines a series of "log paths" that define how ~Syslog-NG handles messages. Log paths consist of sources, destinations and optional filters. Messages arrive from sources and are delivered to destinations.
Sources available include udp and tcp network ports, named pipes, files and UNIX sockets. Destinations include all of these and programs. Filters can use regular expressions, facilities, levels and programs. Log paths can contain multiple sources and destinations.
You can get ~Syslog-NG from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/.
TACACS+ is an AAA protocol for Cisco network equipment. AAA stands for Authentication, Authorisation and Accounting. TACACS+ can be used to provide fine grained control over the commands that can be issued to network equipment. It can also record all the commands that are issued. Cisco published some reference demo code when it established an RFC ([[RFC 1492|http://rfc.net/rfc1492.html]]) for [[TACACS+|http://www.cisco.com/warp/public/614/7.html]]. The opensource versions are based on this code.
You can get a version of TACACS+ from http://www.shrubbery.net/tac_plus/. This version includes acls to control which devices can be controlled by an account. I use this version with PAM patches from http://web.tiscalinet.it/macchese/PAM/tacacs.html so I can slave authentication off other systems. The PAM patches were intended for the original TACACS+ demo code published by Cisco but they aren't terribly hard to integrate by hand with the version from shrubbery.net.
[[Thrulay|http://thrulay.sourceforge.net]] is a tool for testing available network bandwidth. It is a latter development of [[iperf]] and was originally called iperf2. You run thrulayd on one host, uses port 5003 by default, and thrulay on another. It will then test the network path between the two hosts by sending traffic between them. Thrulay can test bi and unidirectional TCP and UDP bandwidth of a link. The TCP tests report bandwidth, round trip delay and jitter. The UDP tests report round trip delay, jitter, packet loss, duplication and reordering, You specify the desired bandwidth. All Thrulay results are reported at the client.
{{{
$ /usr/local/sbin/thrulayd
$ thrulay -t 10 127.0.0.1
# local window = 4194304B; remote window = 4194304B
# block size = 65536B
# MTU: 16436B; MSS: 16332B; Topology guess: Linux Loopback device
# MTU = guessed out of MSS as in RFC 879; MSS = getsockopt(TCP_MAXSEG)
# test duration = 10s; reporting interval = 1s
# delay (median) and jitter (interquartile spread of delay) are reported in ms
#(ID) begin, s end, s Mb/s RTT delay,ms jitter
( 0) 0.000 1.000 6913.558 8.794 3.359
( 0) 1.000 2.000 6759.618 9.310 3.318
( 0) 2.000 3.000 6863.053 9.186 3.210
( 0) 3.000 4.000 6659.564 8.810 2.847
( 0) 4.000 5.001 7087.585 8.920 2.771
( 0) 5.001 6.000 6799.462 9.161 3.001
( 0) 6.000 7.000 7067.028 9.021 2.723
( 0) 7.000 8.000 6832.936 8.951 2.482
( 0) 8.000 9.000 6780.492 9.321 2.321
( 0) 9.000 10.000 6961.503 8.923 2.178
#(**) 0.000 10.000 6872.234 8.999 3.225
$ thrulay -t 10 -u10k 127.0.0.1
Delay: 0.159ms
Loss: 0.000%
Jitter: 0.016ms
Duplication: 0.000%
Reordering: 0.000%
}}}
URL:http://www.tiddlytools.com/
This entry just provides a list of the network management tools discussed on this site.
I've just finished a project that uses the Perl net-snmp library and ~SNMPv3 and I found using the //localized// keys tricky and not well documented. The trick was that the keys are required to be passed as a packed binary, however the ~EngineIDs are hex strings, without a 0x prefix, so use pack to pack the keys but not the ~EngineIDs.
{{{
my $engine_id = "80000009fe36d257d0d9d1011f770b3fabebb51d69cd4b4f4d";
my $auth_localized_key = "017e0d6000c6d41bad62a02fb37df069";
my $priv_localized_key = "1eee8a9a9ad7c697d3a685751e4249a2";
my %options =
{
...
SecEngineId => $engine_id,
AuthLocalizedKey => pack('H*', $auth_localized_key),
PrivLocalizedKey => pack('H*', $priv_localized_key),
}
my $session = SNMP::Session->new(%options);
}}}
In the above example I have left out the options that aren't to do with ~EngineIDs or keys.
You can use [[snmpkey]] to generate the //localized// keys.
<!--{{{-->
<div class='toolbar' macro='toolbar closeTiddler closeOthers +editTiddler > fields syncing permalink references jump'></div>
<div style='float:right;' macro='tiddler BookMarkTiddler'></div>
<div class='title'><span macro='view title'></span></div>
<div class='subtitle'><span macro='view modifier link'></span>, <span macro='view modified date'></span> (<span macro='message views.wikified.createdPrompt'></span> <span macro='view created date'></span>)</div>
<div class='tagging' macro='tagging'></div>
<div class='tagged' macro='tags'></div>
<div class='viewer' macro='view text wikified'></div>
<div class='tagClear'></div>
<!--}}}-->
{{center{
[img[Visitor Map|http://www.maploco.com/vmap/1709538.png][http://www.maploco.com/view.php?id=1709538]]
[[Create your own visitor map!|http://www.maploco.com/]]
}}}
<<showUpdates excludeTag:excludeTimeline maxEntries:8>>
[[Wireshark|http://wireshark.sourceforge.net]] is a network sniffer and protocol decoder. It use to be called Ethereal. With Wireshark you can capture traffic from (mainly Ethernet) network interfaces and then display a detailed description of what happened in a graphical screen. It can display packet traces captured with other tools such as tcpdump and snoop and display previously captured traces, so you can capture packets on headless or remote machines and decode them on your desktop, or get your clients to capture a packet trace of a problem and send it to you.
It has extensive, but somewhat confusing, options for selecting what is displayed (and captured) which is essential for this kind of work. There is however a graphical filter (behind the +Expression button) builder that I make extensive use of when I am using Wireshark. There are some analysis tools for the capture and one that I use all the time is "Follow TCP Stream", by selecting one packet from a TCP stream you can restrict the packets displayed to just packets from this stream.
| [img[Wireshark Screen|images/wireshark2.png]] |
| Sample Wireshark Screen |
| borderless |k
I recently discovered the __{{{C}}}__{{{onversations}}} and __{{{E}}}__{{{ndpoints}}} tools in the __{{{S}}}__{{{tatistics}}} menu. These two tools let you pick out all the TCP and UDP ports used by an application, great for reverse engineering protocols for firewall rules.
Wireshark is included in many distros and packages are available for almost everything else, including Windows.
The [[Hobbit]] is now called Xymon, due to objections from the 'Lord of the Rings' people.
amavis-stats produces graphs from amavis logs. amavis is an opensource virus and spam filtering framework. It graphs clean emails, spam emails and infected emails by virus.
| [img[Sample amavis-stats Graph|images/amavis-stats.jpg]] |
| Sample amavis-stats Graph |
|borderless|k
You can get the latest version from this site http://osx.topicdesk.com/. Just search for amavis-stats.
Bb-event is my system to translate events into Big Brother alarms. It is currently under development and I have yet to release it.
cricketBB is a system to convert [[Cricket]] monitor thresholds into [[BigBrother]] status messages. This provides a link between [[Cricket]], a performance management system, and [[BigBrother]] a fault management system. You can read about it [[here|projects/cricketBB]].
cricketBB has been updated to the latest version (1.03). It fixes a problem with values not changing once a threshold is passed and lets you specify tests on a per target basis.
drraw lets you define graphs, graph templates and dashboards of graph templates of RRD data in from a CGI interface. A dashboard is a display of large thumbnails of multiple graphs of multiple devices. ddraw is very useful for drawing graphs of data collected by other applications or for helping you making your own.
You can get ddraw from http://web.taranis.org/drraw/.
[[gnokii|http://www.gnokii.org/]] is a user space driver and a set of command line tools for controlling mobile phones via a RS232 or USB serial connection. It can be used to send and receive SMS messages, as well as do many other phone functions. It has an API with bindings for many languages.
[[iperf|http://iperf.sourceforge.net/]] is a tool for testing available network bandwidth. To use it you run iperf in server mode on one host, it uses port 5001 by default, and in client mode on another. It will then test the network link between the two hosts by sending traffic between the hosts. It can perform uni and bidirectional TCP tests and unidirectional UDP tests.The TCP tests report amount of data transferred and the available bandwidth. The UDP tests report bandwidth, jitter and lost packets.
The server running on one host ...
{{{
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 256 KByte (default)
------------------------------------------------------------
[ 4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 58025
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 2.54 GBytes 2.18 Gbits/sec
}}}
The client running on another (or in this case the same) host ...
{{{
$ iperf -c 127.0.0.1
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 3] local 127.0.0.1 port 58025 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.54 GBytes 2.18 Gbits/sec
}}}
We recently found out, because it became an issue, that the current iperf payload is a running series of numbers 0 .. 9 repeated over and over.
There are several other similar tools such as [[Thrulay]], netperf, nettest, nuttcp and ttcp. Iperf is in most Linux distros, so you probably already have it. You can find an in depth tutorial on iperf [[here|http://openmaniak.com/iperf.php]].
Nacbackup is a small script I wrote to backup Cisco NAC Appliances to a remote server. You can learn more about it [[here|projects/nacbackup/]].
ntop is a bit like a protocol analyser (think Fluke Network Assistant) running on your server. Originally it collected stats on local traffic on your server's network interfaces now it collects stats from [[sFlow]] or [[NetFlow]] or remote ntop probes.
| [img[ntop Sample Page|images/ntop_vista.jpg]] |
| ntop Sample Page (from ntop.org) |
|borderless|k
You can get ntop from http://www.ntop.org/
[[snmpkey|http://search.cpan.org/~dtown/Net-SNMP-5.2.0/snmpkey.PL]] is a program that generates ~SNMPv3 //localized// keys from passwords and ~EngineIDs. //Localized// keys combine the password and the ~EngineID. ~SNMPv3 uses the //localized// keys to access agents. If you provide a password a //localized// key has to be produced before you can get what you want.
Why should I care? Using a //localized// key speeds up ~SNMPv3 access in two ways. Firstly the management system doesn't have to produce the key, which takes time. Secondly the ~EngineID of the agent does not have to be probed for, which means an additional network transaction.