About Wayne Wan

Travel & Sports & Work
Nov 28

How to install Hadoop on windows7

    During my installing of hadoop on my laptop which was based on windows7, I encountered some problems which cost me couple of hours, so I plan to write it down.

    The standard process was  :

  1. install Cygwin(must enable openSSH feature when installing).
  2. download Hadoop package from official site, and extract it.(PS: in my laptop, I must extract it in the directory which cygwin located, or hadoop namenode would not be started. )
  3. set environment variables: JAVA_HOME, HADOOP_INSTALL, add %HADOOP_INSTALL%/bin to PATH.
  4. open %HADOOP_INSTALL%/conf/hadoop-env.sh, add "export JAVA_HOME=***", add "export HADOOP_INSTALL=**".
  5. then open Cygwin terminal, type "hadoop version", if no error found ,then it's ok. if error found , check your HADOOP_INSTALL, and PATH setting.(PS: be ware of the JAVA_HOME, it is recommended that this path should not include " "(space)).
  6. modity core-site.xml, hdfs-site.xml, mapred-site.xml using below content:
    <?xml version="1.0"?>
    <!– core-site.xml –>
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost/</value>
    </property>
    </configuration>
    <?xml version="1.0"?>
    <!– hdfs-site.xml –>
    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>
    <?xml version="1.0"?>
    <!– mapred-site.xml –>
    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>localhost:8021</value>
    </property>
    </configuration>
  7. configuring SSH using command: ssh-host-config , select not create private privilege account(may not the same word, but seems like it.).
    then generate a new SSH key (so we can login without password, the same as hadoop running on it) .
    % ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    % cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    Test with : ssh localhost ( before we can login , we must start the cygwin ssh service firstly. )
  8.  initialize hadoop namenode using :  hadoop namenode -format. if you can find "tmp/hadoop-*/dfs/name" in the directory which hadoop located, then congratulations.
  9. run start-all.sh (it will start namenode, jobtracker, secondary namenode, datanode). using jps command to verify if these deamon process were started successfully. also can check it by http://localhost:50030/ for the jobtracker, http://localhost:50070/ for the namenode.

    In my laptop, I must change the cygwin service configuration, I must change the service logon role. (using the account which was in administrator group)

Jul 14

How to solve “com.ibm.etools.rmic.RMICException” when do serviceDeploy

    Today, when I perform the serviceDeploy on our mediation project I encountered a strange problem.I could serviceDeploy successfully manually, but not able to do it by hudson. When I used hudson to generate ear package automatically, I got deployment failed from the console, then I tried to find more details, after hours and hours later, I found there was an parameter I could used to trace the problem like the parameter of serviceDeploy.bat has.(The parameter was 'keep' ).  After  I turned on this switch, I found "com.ibm.etools.rmic.RMICException", the whole stacktrace were: Continue reading

Feb 24

[转]WebSphere Concepts: Cell, Node, Cluster, Server…

Quick post… If you are not familiar with WebSphere at first you might get confused with its concepts: cell, deployment manager, node, node agent, cluster, server, …

First of all, lets start with the concept of a Cell:

A Cell is a virtual unit that is built of a Deployment Manager and one or more nodes. I guess a picture will help making things clearer: Continue reading

Jan 12

Websphere Message Broker7中配置DB2 JDBCProvider

    JDBCProvider向Websphere Message Broker(WMB)中需要操作数据库的node提供数据库连接服务,JDBCProvider作为一个服务被注册到指定的broker中。

     这里需要涉及到的几条命令有:

  1. mqsideleteconfigurableservice, 用于删除一个可配置的服务对象。
    例如:mqsideleteconfigurableservice MB7BROKER -c JDBCProviders -o SIMPLERROUTEDB
    表示从MB7BROKER这个broker中删除type为JDBCProviders的服务(configurable service)中的名为SIMPLERROUTEDB可配置对象(configurable service object).
  2. mqsicreateconfigurableservice,用于创建一个可配置的服务对象(configurable service)
    例如: Continue reading
Jul 10

英文俚语收藏

sporting house 妓院(不是“体育室”) 

dead president 美钞(上印有总统头像)(并非“死了的总统”) 

lover 情人(不是“爱人”)

busboy 餐馆勤杂工(不是“公汽售票员”)

busybody 爱管闲事的人(不是“大忙人”)

dry goods (美)纺织品;(英)谷物(不是“干货”)

heartman 换心人(不是“有心人”) 

mad doctor 精神病科医生(不是“发疯的医生”)

eleventh hour 最后时刻(不是“十一点”) 

blind date (由第三者安排的)男女初次会面(并非“盲目约会”或“瞎约会”) 
 

Continue reading

May 22

linux RPM命令详解

RPM 是 Red Hat Package Manager 的缩写,本意是Red Hat 软件包管理,顾名思义是Red Hat 贡献出来的软件包管理;在Fedora

、Redhat、Mandriva、SuSE、YellowDog等主流发行版本,以及在这些版本基础上二次开发出来的发行版采用;
RPM包里面都包含什么?里面包含可执行的二进制程序,这个程序和Windows的软件包中的.exe文件类似是可执行的;RPM包中还包括程序运行时所
需要的文件,这也和Windows的软件包类似,Windows的程序的运行,除了.exe文件以外,也有其它的文件;
一个RPM
包中的应用程序,有时除了自身所带的附加文件保证其正常以外,还需要其它特定版本文件,这就是软件包的依赖关系;依赖关系并不是Linux特有的,
Windows操作系统中也是同样存在的;比如我们在Windows系统中运行3D游戏,在安装的时候,他可能会提示,要安装Direct 9
;Linux和Windows原理是差不多的; Continue reading

May 22

rhE Linux4下NFS服务器安装和共享配置

    如果在安装系统的时候是全安装,则NFS是会被直接装好的,若无法确定系统是否已经安装NFS服务器,可用用rpm命令查看是否安装portmap跟nfs-utils,如下:rpm -qa | grep portmap ,rpm -qa | grep nfs ,更精确的查询是:rpm -q portmap,rpm -q nfs-utils。若提示not installed则必须自己手动安装了。 Continue reading

May 22

误用jakata commons-beanutils引起bug分析

    项目里面大量的使用到了jakata的commin包,包括beanutils这个包,主要是用来copy bean property,以前一直用的很好,也没有去深究它内部的实现机制,评自己猜想就只是运用反射去取得属性的值,然后设置值吧。接着就出现了现在遇到的问题,项目中一个vo属性的类型为Byte包装类型,这种属性在实例化的时候默认为null,这个特性被我用到了业务中,也就是我会通过判断这个属性是否为null来实现某些逻辑,结果发现这个属性竟然会在没有任何人动的情况下自己被赋值为0,然后就导致依赖这个属性的业务逻辑错掉,debug一步一步的跟踪发现只要一经过beanutils,copy出来的bean的这个属性自动变为0了,以为是beanutils的bug,找来源码看了看,发现BeanUtils中其实是调用了BeanUtilsBean的copyProperties方法,BeanUtilsBean中实现了一个简单的工厂模式,将的一个实例绑定在当前线程中,有点儿类似ThreadLocal,也就是当前线程中所有调用都会采用同一机制进行copy,这里说到copy的机制其实就是指Converter,对于各种类型都有一个特定的Converter,包内部提供了一些jdk内部类型的Converter,每种类型Converter都有可以提供一种默认值,而在BeanUtils中对于Byte,Integer等数值类型提供的默认值是:0 ,所以就会出现把Byte类型值为null转换成了0。其实这么做也有他的好处,可以避免出现烦人的NullPointerExceprtion。知道原因,解决办法就很容易想到了,自己重新初始化一下BeanUtilBean中的ByteConverter。